A gesture recognition mapping method and system based on robot vision

Abstract

The application discloses a gesture recognition mapping method and system based on robot vision, and the method comprises the following steps: collecting two-dimensional coordinates of each key point of a hand; performing angle calculation on each finger joint based on vector geometry to obtain continuous bending degree values of five fingers, and introducing a dynamic calibration mechanism to eliminate individual hand size differences; then, combining preset logic rules and a dynamic threshold adjustment mechanism, the bending degree vector is determined as a specific gesture semantic; finally, the bending degree value is converted into a target control parameter of a robot actuator through a calibrated mapping function, so that continuous and smooth mapping from a gesture to a mechanical hand action is realized. The application realizes real-time gesture recognition and accurate reproduction with low computing power and high robustness without relying on a depth camera, effectively overcomes the problems of high hardware cost, poor individual adaptability and stiff action mapping in the prior art, and can be widely applied to the fields of humanoid robots, virtual reality and other fields requiring fine gesture interaction.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and intelligent human-computer interaction technology, and in particular relates to a low-computing-power, high-robust gesture recognition method based on monocular two-dimensional vision and its application in robot motion control. Background Technology

[0002] In the field of human-machine interaction technology, gestures, as an intuitive and natural form of communication, are receiving increasing attention. With the rapid development of humanoid robot technology, enabling robots to understand and imitate human gestures has become a key breakthrough in improving the human-machine collaborative experience. Whether it's service robots learning human gestures, educational robots following instructions for gesture teaching, or social robots establishing emotional connections with users through body language, the ability to imitate gestures directly determines the naturalness and approachability of human-machine interaction. However, to achieve accurate reproduction of complex gestures by humanoid robots, the first hurdle to overcome is the fundamental technological barrier of gesture recognition.

[0003] Current mainstream gesture recognition solutions diverge significantly in their technological approaches. One type relies on depth cameras or multi-view stereo vision systems to calculate gestures by acquiring three-dimensional spatial information of the hand. While this approach can theoretically achieve high recognition accuracy, its high hardware costs and demanding computational requirements hinder its large-scale deployment on cost-sensitive or resource-constrained humanoid robot embedded platforms, limiting its widespread application in consumer products. The other type is based on ordinary 2D color cameras, employing traditional image processing and pattern recognition methods. However, this approach faces numerous challenges in practical applications: variations in lighting conditions, interference from complex backgrounds, and individual differences in user hand size and shape can all lead to significant fluctuations in recognition performance. Existing 2D vision methods often perform well in specific controlled environments, but once transferred to real-world scenarios with varying lighting and cluttered backgrounds, their accuracy and stability become difficult to guarantee, resulting in frequent misjudgments and frame drops.

[0004] Furthermore, existing solutions generally fall short in addressing user diversity. Users of different ages and hand shapes exhibit significant differences in finger bending range and habitual postures, and traditional recognition methods using fixed thresholds or single templates struggle to cover these individual differences, leading to system malfunctions or misjudgments for some users. When gesture recognition technology is applied to the imitation control of humanoid robots, the challenges are further exacerbated. This problem is particularly prominent in scenarios where humanoid robots are open to the general public, becoming one of the bottlenecks restricting the widespread application of the technology. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a gesture recognition and mapping method and system based on robot vision. By optimizing key point feature extraction and angle calculation, combined with dynamic normalization and threshold determination, individual differences and environmental interference are eliminated, enabling real-time and robust gesture recognition. The recognition results can be smoothly mapped to the robot execution end, improving the naturalness and response speed of human-computer interaction.

[0006] Specifically, the technical solution provided by this invention is as follows: A gesture recognition and mapping method based on robot vision includes the following steps: S1. Acquire user hand images using a monocular 2D color camera, and use a hand key point detection algorithm to obtain a set of two-dimensional coordinates of multiple key feature points of the hand in real time; S2. Based on the two-dimensional coordinate set, the bending angle of each finger joint is calculated, and the bending angles of multiple joints contained in each finger are accumulated to obtain the total bending angle of the finger; then, the total bending angle is mapped to a preset standardized range through linear normalization, and a continuous bending value representing the degree of finger bending is output. S3. Based on the preset gesture logic rules, combine and determine the continuous bending values ​​of the five fingers, and output the corresponding gesture semantics. S4: Through the pre-calibrated mapping relationship, the continuous curvature value is converted into the target control parameter of the robot actuator, driving the robot actuator to smoothly reproduce the user's gesture in real time.

[0007] Furthermore, the multiple key feature points of the hand mentioned in step S1 include: Key point 0 on the wrist: the center point of the wrist joint; The key points of the thumb, from 1 to 4, are as follows: the thumb carpal joint, the thumb metacarpophalangeal joint, the thumb interphalangeal joint, and the thumb tip; The key points of the index finger are 5 to 8, in order: the metacarpophalangeal joint of the index finger, the proximal interphalangeal joint of the index finger, the distal interphalangeal joint of the index finger, and the fingertip of the index finger; The key points of the middle finger are 9 to 12, in order: the metacarpophalangeal joint of the middle finger, the proximal interphalangeal joint of the middle finger, the distal interphalangeal joint of the middle finger, and the fingertip of the middle finger; Key points 13 to 16 of the ring finger are, in order: the metacarpophalangeal joint of the ring finger, the proximal interphalangeal joint of the ring finger, the distal interphalangeal joint of the ring finger, and the fingertip of the ring finger; Key points 17 to 20 on the little finger are, in order: the metacarpophalangeal joint of the little finger, the proximal interphalangeal joint of the little finger, the distal interphalangeal joint of the little finger, and the fingertip of the little finger.

[0008] Furthermore, the method for calculating the bending angle of each finger joint in step S2 is as follows: for any joint point B and its two adjacent key points A and C, construct a vector starting from B. and The cosine of the included angle is calculated using the vector dot product formula, and then the bending angle of joint B is obtained. i : ,

[0009] Where cos i The calculation results need to be truncated to the interval [-1, 1].

[0010] Furthermore, the total bending angle of each finger in step S2 is calculated as follows: Total thumb flexion angle Angle_thumb = i 1+ i 2, i 1 represents the joint bending angle calculated from key points 0, 1, and 2. i 2 represents the joint bending angle calculated from key points 1, 2, and 3; Total bending angle of index finger Angle_index = i 5+ i 6+ i 7, of which i 5. i 6. i 7 represents the joint bending angles calculated from key points (0,5,6), (5,6,7), and (6,7,8), respectively. Total bending angle of the middle finger Angle_middle = i 9+ i 10 + i 11 ,in i 9. i 10 , i 11 The joint bending angles are calculated from the key points (0,9,10), (9,10,11), and (10,11,12), respectively. Angle_ring = total bending angle of the ring finger i 13 + i 14 + i 15 ,in i 13 , i 14 , i 15 The joint bending angles are calculated from the key points (0,13,14), (13,14,15), and (14,15,16), respectively. Angle_pinky = Total bending angle of little finger i 17 + i 18 + i 19 ,in i 17 , i 18 , i 19 The joint bending angles are calculated from the key points (0,17,18), (17,18,19), and (18,19,20), respectively.

[0011] Furthermore, in step S2, the formula for calculating linear normalization is:

[0012] in Angle The total bending angle of the finger. Angle min and Angle max These are the typical values ​​of the total bending angle under preset fully bent and fully straightened states, respectively. The typical value of the total bending angle for the thumb, Angle min Take 150°, Angle max Take 300°; for the index, middle, ring, and little fingers, Angle min Take 270°, Angle max Alternatively, at the start of the interaction, guide the user to perform two baseline actions: fully extending their palm and fully clenching their fist. Collect and record the total bending angle of each finger at these times, and use these as personalized values ​​for the user. Angle max and Angle min .

[0013] Furthermore, in step S3, the gesture logic rules include at least one of the following determination rules: If the continuous bending value of all five fingers is not greater than the threshold T, it is judged as a "stone" gesture. If the continuous bending value of all five fingers is greater than the threshold T, it is determined to be a "cloth" gesture. If only the index and middle fingers have a continuous bending value greater than the threshold T, it is judged as a "scissors" gesture. If only the continuous bending value of the thumb is greater than the threshold T, it is judged as a "thumbs up" gesture.

[0014] Furthermore, the threshold is a preset static threshold. The conditions of each gesture logic rule are checked in turn. If multiple rules are met at the same time, the final gesture is determined by priority sorting or confidence weighting. If none of them are met, it is determined to be an unknown or transitional gesture.

[0015] Preferably, the threshold adopts a dynamic threshold adjustment mechanism; The dynamic threshold adjustment mechanism employs a calibration-based static personalized threshold method: it collects the average value of the bending degree of each finger when the user makes a "stone" gesture. And the average degree of bending of each finger when making the "cloth" gesture. ; Calculate personalized thresholds for each finger ,in Indicates the first i The threshold corresponding to the root finger; Alternatively, the dynamic threshold adjustment mechanism employs an adaptive threshold method based on real-time statistics, including: Sliding window statistics: Maintain a queue of finger flexion values ​​for the most recent N frames, calculate the quantile of each finger flexion in real time, and use the specified quantile as the threshold for determining the flexion / straightening of that finger. Extreme value tracking update: The maximum and minimum bending of each finger are tracked in real time using an exponential moving average, and the midpoint between the two is used as the real-time threshold.

[0016] Further, in step S4, the method for establishing the mapping relationship includes: for each finger, selecting multiple representative bending states, recording the corresponding actual servo angles, and forming a calibration point set; establishing a mapping function between bending degree and servo angle through linear interpolation, polynomial fitting, or cubic spline interpolation, and pre-generating a lookup table.

[0017] A gesture recognition mapping system based on the above method, the system comprising the following modules: The image acquisition module uses a monocular 2D color camera to capture image frames of the user's hand in real time; The data processing module is used to preprocess the acquired image frames and call the built-in hand key point detection algorithm to output a set of two-dimensional coordinates of multiple key feature points of the hand. The gesture recognition module receives the two-dimensional coordinate set, obtains the bending angle of each finger joint through geometric calculation, accumulates the total bending angle of each finger, and then outputs a continuous bending degree value representing the degree of finger bending after linear normalization. The gesture recognition module has a built-in dynamic threshold adjustment mechanism and gesture logic rule library, which is used to adaptively optimize the judgment threshold according to the user's hand features and combine the continuous bending degree values ​​of the five fingers to judge and output the corresponding gesture semantics. The motion control module has a pre-stored mapping relationship between curvature and servo angle established through calibration. It receives the continuous curvature values ​​and gesture semantics, converts them into target control parameters for the robot actuator through lookup and interpolation operations, and performs trajectory planning and smoothing on the target control parameters to generate continuous control commands. The interaction management module is connected to the image acquisition module, data processing module, gesture recognition module, and motion control module respectively. It is used to coordinate the working sequence and data flow of each module, manage system startup initialization, interaction state switching, user calibration guidance, and rule base and parameter configuration for multi-mode interaction.

[0018] Compared with the prior art, the present invention has at least the following beneficial effects: At the hardware level, this invention abandons high-cost solutions such as depth cameras or multi-view vision systems, and achieves high-precision gesture recognition by relying solely on a single 2D color camera, which greatly reduces the hardware threshold and overall cost, enabling gesture interaction technology to be deployed in various consumer robot products and embedded platforms in a more economical way.

[0019] In terms of recognition accuracy and robustness, this invention effectively eliminates interference caused by changes in lighting, cluttered backgrounds, and differences in user hand size through precise joint angle calculation and multi-level normalization processing. The introduction of a dynamic calibration mechanism enables the system to obtain the user's personalized bending extreme value, ensuring that the normalization process truly adapts to the physiological characteristics of each user, significantly improving the recognition accuracy for users of different ages, genders, and hand shapes.

[0020] At the robot control level, this invention achieves high-fidelity reproduction of user gestures to robot movements through precise mapping calibration and smooth trajectory planning. The pre-established mapping relationship between curvature and servo angle ensures accurate conversion, while multiple smoothing mechanisms such as trajectory interpolation, low-pass filtering, and velocity acceleration limiting make the robot's movements smooth and natural, completely eliminating the stiffness and mechanical impact of traditional solutions, and enhancing the realism and friendliness of the interaction.

[0021] From a system integration perspective, this invention adopts a modular design, forming a complete closed loop from image acquisition to motion control. The interaction management module, acting as the scheduling hub, can flexibly support switching between multiple application modes, such as rock-paper-scissors games, gesture teaching, and action imitation. Simultaneously, the system's real-time processing capability within milliseconds ensures a seamless interaction, providing users with a smooth and natural human-computer interaction experience.

[0022] In summary, this invention achieves high-precision, robust, and personalized gesture recognition and smooth mapping with low-cost hardware, providing a practical technical solution for fine gesture interaction in fields such as humanoid robots, service robots, and virtual reality, and has significant market application value and promotion prospects. Attached Figure Description

[0023] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0024] Figure 1 This is a schematic diagram of the gesture recognition mapping method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of key hand feature points provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the composition of the gesture recognition mapping system modules provided in an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, other embodiments obtained by those skilled in the art without creative effort are all within the scope of protection of the present invention.

[0026] Example 1 This embodiment provides a gesture recognition mapping method based on robot vision, such as... Figure 1 As shown, the method mainly includes the following steps: 1. Two-dimensional coordinate acquisition of key hand points This step aims to acquire structured feature data of the user's hand using a monocular vision sensor, laying the foundation for subsequent gesture analysis and recognition. Specifically, this embodiment uses a monocular 2D color camera as the image acquisition device to capture RGB image frames containing the user's hand in real time. The camera needs to have appropriate resolution and frame rate to ensure clear capture of hand details in dynamic interactive scenarios, with a resolution of no less than 640×480 and a frame rate of at least 30 frames per second to meet real-time requirements.

[0027] After acquiring the raw image data, the system's built-in hand keypoint detection algorithm is used to process each frame of the image. This algorithm is built on a deep learning model and employs a lightweight convolutional neural network architecture, enabling real-time inference on embedded platforms or resource-constrained robotic systems. As a preferred implementation, a detection model similar to MediaPipe Hand Landmarker can be used. This model captures multi-scale spatial features through a stacked hourglass structure, ultimately outputting the positional information of 21 keypoints on the hand, which cover the complete skeletal structure of the hand.

[0028] like Figure 2 As shown, according to the standardized definition of hand key points, they specifically include: Key point 0 on the wrist: the center point of the wrist joint; The key points of the thumb, from 1 to 4, are as follows: the thumb carpal joint, the thumb metacarpophalangeal joint, the thumb interphalangeal joint, and the thumb tip; Key points 5 to 8 of the index finger are, in order: metacarpophalangeal joint of the index finger, proximal interphalangeal joint of the index finger, distal phalangeal joint of the index finger, and fingertip of the index finger; The key points of the middle finger are 9 to 12, in order: the metacarpophalangeal joint of the middle finger, the proximal interphalangeal joint of the middle finger, the distal phalangeal joint of the middle finger, and the tip of the middle finger; Key points 13 to 16 of the ring finger are, in order: the metacarpophalangeal joint of the ring finger, the proximal interphalangeal joint of the ring finger, the distal phalangeal joint of the ring finger, and the tip of the ring finger; Key points 17 to 20 on the little finger are, in order: the metacarpophalangeal joint of the little finger, the proximal interphalangeal joint of the little finger, the distal phalangeal joint of the little finger, and the tip of the little finger.

[0029] The keypoint detection algorithm outputs a set of two-dimensional coordinates of 21 key feature points of the palm and fingers, in the form {( x 0, y 0), ( x 1, y 1), ..., ( x 20 , y 20 The coordinate set, where each coordinate value is in pixels of the image, fully describes the spatial configuration of the hand in the current frame, including posture information such as finger bending, extension, and opening / closing. In practical applications, to further improve the stability and consistency of the coordinate data, temporal filtering can be performed on the coordinates of multiple consecutive frames, such as using Kalman filtering or sliding window averaging, to eliminate coordinate fluctuations caused by slight hand tremors or detection noise, and obtain smooth keypoint trajectories.

[0030] 2. Calculation and normalization of finger joint angles After collecting the two-dimensional coordinates of 21 key points on the hand, the data were analyzed to quantify the degree of bending of each finger, providing a quantitative basis for subsequent gesture determination.

[0031] 2.1 Vectorized Modeling For any joint where the bending angle needs to be calculated, select that joint point and its two adjacent key points as vertices. Taking the proximal interphalangeal joint of the index finger (key point 6) as an example, its adjacent points are the metacarpophalangeal joint point 5 and the distal interphalangeal joint point 7. Let the coordinates of the three points be A( x 5, y 5) B ( x 6, y 6) C ( x 7, y 7) To calculate the bending angle of joint B, two vectors need to be constructed with B as the starting point, namely vector BA and vector BC.

[0032] The formula for calculating vector BA is:

[0033] The formula for calculating vector BC is:

[0034] These two vectors point to the two sides of the bones at joint B, and the angle between them is the bending angle of joint B. According to the definition of the dot product of vectors, the cosine of the angle between the two vectors can be calculated by the following formula:

[0035] Where the numerator is the dot product of two vectors:

[0036] The denominator is the product of the magnitudes of the two vectors:

[0037]

[0038] Due to potential numerical errors in the calculation process, cos i The calculated result slightly exceeds the theoretical range of [-1, 1], therefore, cos... i The angle is truncated and confined to the interval [-1, 1]. Then, the angle value is obtained using the inverse cosine function. i that angle i This refers to the bending angle of joint B. When the finger is fully extended, this angle is close to 180°; when the finger is fully bent, this angle approaches a smaller value, even approaching 0°. Using the same geometric method, each joint of each finger can be calculated individually. Specifically: For the thumb, the bending angles of joint 1 (key point 1) and joint 2 (key point 2) need to be calculated. The angle of joint 1 is calculated from key points 0, 1, and 2, and the angle of joint 2 is calculated from key points 1, 2, and 3.

[0039] For the index finger, the bending angles of joint 5 (key point 5), joint 6 (key point 6), and joint 7 (key point 7) need to be calculated. The angle of joint 5 is calculated from key points 0, 5, and 6; the angle of joint 6 is calculated from key points 5, 6, and 7; and the angle of joint 7 is calculated from key points 6, 7, and 8.

[0040] For the middle finger, the bending angles of joints 9, 10, and 11 need to be calculated, which are obtained from key points (0,9,10), (9,10,11), and (10,11,12), respectively.

[0041] For the ring finger, the bending angles of joints 13, 14, and 15 need to be calculated, which are obtained from key points (0,13,14), (13,14,15), and (14,15,16), respectively.

[0042] For the little finger, the bending angles of joints 17, 18, and 19 need to be calculated, which are obtained from key points (0,17,18), (17,18,19), and (18,19,20), respectively.

[0043] 2.2 Cumulative Normalization After obtaining the bending angles of each joint of each finger, these discrete angle values ​​need to be integrated into a single value that can characterize the overall bending degree of the entire finger. This embodiment uses a summation method to add the bending angles of multiple joints on a finger to obtain the total bending angle of the finger.

[0044] Taking the index finger as an example, its total bending angle Angle_index = i 5+ i 6+ i 7, of which i 5. i 6. i 7 represents the calculated angles of the base joint, proximal interphalangeal joint, and distal interphalangeal joint of the index finger, respectively. Similarly, the total bending angle of the thumb is Angle_thumb = i 1+ i 2; Middle finger Angle_middle = i 9+ i 10 + i 11 ; Ring finger Angle_ring = i 13 + i14 + i 15 ;Little finger Angle_pinky= i 17 + i 18 + i 19 .

[0045] The total bending angle visually reflects the overall degree of finger flexion: when the fingers are fully extended, the angles of each joint are close to 180°, resulting in a larger total bending angle; when the fingers are fully clenched, the angles of each joint decrease significantly, and the total bending angle decreases accordingly. However, due to differences in hand size and bone proportions among different users, even when making the same gesture, the absolute angle value may vary. For example, the difference in hand size between children and adults can lead to inconsistent ranges of angle values ​​calculated for the same gesture. To eliminate the influence of such individual differences on subsequent judgments, the total bending angle needs to be normalized, mapping it to a standardized numerical range.

[0046] Normalization requires a pre-defined angle mapping interval [Angle] min Angle max This range should cover the minimum and maximum total bending angles that all users may experience during the process of fully bending and fully straightening their fingers. By statistically analyzing a large amount of hand data from users of different ages and genders, a universally applicable empirical range can be determined. As a preferred implementation, the angle mapping range for the thumb is set to [150°, 300°], and the mapping range for the other four fingers is set to [270°, 540°]. The lower limit of this range (e.g., 270°) corresponds to the typical value of the total bending angle when the fingers are fully bent, and the upper limit (e.g., 540°) corresponds to the typical value of the total bending angle when the fingers are fully straightened.

[0047] The normalization calculation uses a linear mapping formula. For any finger's total bending angle Angle, its normalized bending value Finger_Flex can be calculated using the following formula:

[0048] This formula linearly compresses the original angle value to the range [0, 1]. The calculated result, Finger_Flex, is the quantified value of the finger's degree of bending, with a clear meaning: 0 represents the finger fully bent (like a clenched fist), and 1 represents the finger fully extended (like an open palm). If the calculated Angle value exceeds the preset range, it is truncated: when Angle... <Angle min When the value is greater than the value of Angle, set Finger_Flex to 0; when the value is greater than the value of Angle, set Finger_Flex to 0. maxWhen that happens, set Finger_Flex = 1.

[0049] Through the above normalization process, the absolute angle differences caused by differences in hand size among different users are effectively eliminated, and the obtained curvature values ​​are consistent across individuals. For example, whether it is an adult or a child, when making the "stone" gesture, the curvature of the five fingers should be close to 0; when making the "cloth" gesture, the curvature of the five fingers should be close to 1.

[0050] To further enhance the adaptability of normalization, a dynamic calibration mechanism can be introduced. At the start of each interaction, the user is guided to perform two baseline actions: fully extending the palm and fully clenching the fist. The system collects and records the total bending angle of each finger in real time, which serves as the user's personalized Angle. max and Angle min This dynamic calibration can more accurately adapt to individual differences, and is especially suitable for applications that require high-precision gesture reproduction, such as humanoid robot mimicry control.

[0051] Ultimately, the output of this step is the bending degree values ​​of each of the five fingers, forming a five-dimensional vector [F_thumb, F_index, F_middle, F_ring, F_pinky], where each component is a continuous real number in the interval [0, 1]. This vector fully quantifies the bending features of the current hand posture, providing accurate input data for subsequent gesture semantic determination. The entire processing must be completed within milliseconds to ensure smooth real-time interaction.

[0052] 3. Gesture semantic determination This step aims to map the obtained five-finger bending vector into discrete gesture semantic categories according to preset logic.

[0053] 3.1 Preset Gestures First, a set of logical rules for basic hand gestures is defined. These rules are based on the combination of finger bends and are determined by comparing their relative magnitudes with a threshold. For ease of description, a threshold parameter is defined. T (Typical value can be 0.5), and an optional finger count function. Examples of rules for determining common gestures are as follows: (1) Gesture “stone” (clenched fist): The bending degree of all five fingers is less than or equal to the threshold T. This rule indicates that all fingers are in a bent state.

[0054] (2) Gesture “cloth” (palm extended): The bending degree of all five fingers is greater than the threshold T. This rule indicates that all fingers are in a straight state.

[0055] (3) Gesture "scissors" (only index and middle fingers are straight): the curvature of the index and middle fingers is greater than the threshold T, and the curvature of the thumb, ring finger and little finger is less than or equal to the threshold T.

[0056] (4) Gesture "thumbs up" (only the thumb is straight): the degree of bending of the thumb is greater than the threshold T, and the degree of bending of the other four fingers is less than or equal to the threshold T.

[0057] (5) Gesture "OK" (thumb and index finger tips touching to form a ring, the other three fingers extended): This gesture is more complex to determine and usually requires consideration of fingertip distance. However, based on curvature, an approximate rule can be set first: the curvature of the thumb and index finger is moderate (close to 0.5), while the other three fingers are straight, that is: the curvature of the thumb and index finger is within the set range (such as 0.3~0.7), and the curvature of the middle, ring, and little fingers is greater than the threshold T. More precise determination requires the introduction of finger distance or angle features.

[0058] (6) Gesture “three” (thumb and little finger straight, others bent): often used to indicate making a phone call or rocking gesture. The rule is: the degree of bending of the thumb and little finger is greater than the threshold T, while the degree of bending of the index, middle and ring fingers is less than or equal to the threshold T.

[0059] The above rules are merely examples; in practical applications, the judgment logic for any gesture can be defined according to requirements. The judgment logic can be represented as a Boolean expression, or implemented using decision trees, lookup tables, or other methods.

[0060] 3.2 Dynamic Threshold Adjustment Mechanism Static thresholding is the most basic implementation method. A global threshold is preset, and the aforementioned logical rules are directly applied. After calculating the curvature vector for each frame, the conditions of each gesture rule are checked sequentially. If multiple rules are satisfied (possibly due to gesture transitions or noise), priority ranking (e.g., prioritizing more complex gestures) or confidence-weighted methods can be used to determine the final output. If no rule is satisfied, the gesture is determined to be unknown or a transitional gesture.

[0061] Static thresholding is simple and easy to implement, but it has significant limitations: while the complete bending or straightening of fingers is standardized after normalization, individual perception thresholds for bending and straightening may still differ. For example, some people habitually bend their fingers slightly, while others straighten them completely. Static thresholding may misjudge normal gestures of some users as transitional states. Therefore, dynamic threshold adjustment needs to be introduced to automatically optimize the judgment threshold based on the current user's hand characteristics, making it more in line with individual habits.

[0062] 3.2.1 Calibration-based static personalized threshold This method requires a brief calibration process before formal interaction. The system guides the user to make several reference gestures, typically extreme gestures (such as "stone" and "cloth"), and records the bending values ​​of each finger at these gestures. The specific steps are as follows: (1) Collect reference values The user makes a "stone" gesture (fully clenched fist), holds it for several frames, and the system records the average value of the bending degree of each finger, denoted as _____. Ideally, these values ​​should be close to 0.

[0063] The user makes a "cloth" (fully extended) gesture, holds it for several frames, and the system records the average value of the bending degree of each finger, which is the mean. Ideally, these values ​​should be close to 1.

[0064] (2) Calculate the personalized threshold Based on the collected reference values, an independent dynamic threshold can be set for each finger. A simple method is to take... F min and F max Midpoint:

[0065] Alternatively, a weighted average or quantiles can be used depending on the actual distribution.

[0066] In subsequent recognition, each finger is judged using its own corresponding threshold. For example, when determining whether the index finger is straight, a comparison is made... and When determining whether the index finger is bent, compare... and The direction is reversed. The logical rules are adjusted accordingly to be based on comparisons using personalized thresholds. If the user does not provide calibration, the system can use the default global threshold T or an average threshold based on user group statistics.

[0067] 3.2.2 Adaptive Threshold Based on Real-Time Statistics For scenarios where display calibration is not possible, an online adaptive algorithm can be used to statistically analyze the distribution of the user's finger curvature in real time and dynamically adjust the threshold.

[0068] (1) Sliding window statistics: Maintain a fixed-length queue to store the finger flexion values ​​of the most recent N frames. Calculate the mean and standard deviation of the finger flexion in real time. Set a threshold of mean ± a certain number of standard deviations, or use quantiles (such as the 30% and 70% quantiles) as the boundary between flexion and straightening. For example, consider the fingers... i threshold T iSet the threshold to the 50th percentile (median) of historical data, or set the bending / straightening classification threshold to the intersection of two Gaussian distributions (assuming a bimodal distribution). (2) Threshold update based on extreme value tracking: Continuously track the maximum and minimum flexion of each finger and slowly decay over time. For example, use exponential moving average updates:

[0069]

[0070] Wherein is the forgetting factor (e.g., 0.99), and the real-time threshold can be the midpoint between the two. This method can adapt to changes in hand posture, but care should be taken to avoid interference from outliers.

[0071] 4. Smooth mapping from gestures to robot movements After completing gesture recognition and obtaining the normalized bend vectors [F_thumb, F_index, F_middle, F_ring, F_pinky] of each finger, this step aims to convert these abstract bend values ​​into target control parameters that can be directly executed by the robot actuator (such as the servo motor of the manipulator), and ensure that the motion process is smooth and natural, so as to achieve real-time and accurate reproduction of the user's gestures.

[0072] 4.1 Calibration Phase: Establishing the mapping relationship between curvature and servo angle Since different models of servos have different turning angle ranges (usually 0~180° or 0~270°), and the mechanical structure of the manipulator (such as linkages and wire harness transmission) may cause a nonlinear relationship between the actual bending angle of the finger and the output angle of the servo, it is necessary to calibrate in advance and establish a mapping function or lookup table between the normalized bending degree and the target angle of the servo.

[0073] (1) Single-point calibration method For each finger, select several representative bending states (e.g., fully extended, fully bent, and several intermediate states), and record the actual angle values ​​of the servo motor in each state by manually controlling the servo or using external measuring tools. The specific steps are as follows: Initialization: Place the robotic arm in its naturally extended position and record the angles of each servo motor at this time. (correspond f =1).

[0074] Full bend calibration: Control the servo motor to fully bend your finger and record the angle of each servo motor. (correspond f =0).

[0075] Intermediate point calibration: To compensate for nonlinearity, 2-3 additional intermediate points can be calibrated, for example...f =0.25, 0.5, 0.75 servo angles , , The calibration method can be as follows: manually adjust the servo motor until the finger presents a roughly corresponding degree of curvature, measure the actual bending angle of the finger using an external protractor or visual method, and record the servo motor angle. Alternatively, observe the finger posture through a preset servo motor angle sequence and select an intermediate state that conforms to visual perception.

[0076] Establish a set of calibration points for each finger. ,in i Number the fingers. j This is the calibration point number.

[0077] (2) Mapping function fitting Based on the calibration points, various methods can be used to establish mapping relationships: Linear interpolation: The simplest and most commonly used method. Linear interpolation is used between two adjacent calibration points.

[0078] in and These are adjacent calibration points. This method can guarantee accuracy when the calibration points are sufficiently close together.

[0079] Polynomial fitting: If the nonlinearity is strong and the number of calibration points is small, a low-order polynomial (such as a quadratic or cubic polynomial) can be used to fit the entire interval.

[0080] Solving the coefficients using the least squares method a , b , c .

[0081] Piecewise nonlinear interpolation, such as cubic spline interpolation, can ensure smooth curves and is suitable for scenarios requiring high precision.

[0082] To facilitate embedded real-time computing, the mapping relationship is usually pre-formed into a lookup table (LUT). For example, with a step size of 0.01, servo angles corresponding to 101 discrete points are generated. In actual use, arbitrary values ​​are obtained by looking up the table and linear interpolation. f The corresponding angle. The formula for generating the lookup table is as follows:

[0083] in N The size of the table (e.g., N=101). The above is the interpolation or fitting function.

[0084] 4.2 Real-time mapping and control During the runtime phase, for each frame of input gesture curvature vector, servo control commands are generated according to the following steps: (1) Lookup / interpolation: For each finger i According to its curvature f i By using a pre-established lookup table or interpolation function, the corresponding target servo angle can be obtained. If a lookup table is used, the index is calculated first. Then linear interpolation:

[0085] (2) Angle limiting: Ensure the target angle is within the mechanical range allowed by the servo (e.g., 0°~180°) to avoid exceeding the limit and damaging the servo.

[0086] (3) Multi-finger coordination: All fingers independently calculate the target angle, forming a five-dimensional target angle vector. .

[0087] 4.3 Smooth Transition and Trajectory Planning Sending the target angle directly to the servo may cause the servo to jump to the target position at maximum speed, resulting in abrupt movement, mechanical shock, or even damage, especially when the hand gesture changes rapidly. Therefore, a smooth control strategy needs to be introduced to make finger movements smooth and natural.

[0088] (1) Trajectory interpolation In each control cycle (e.g., 20ms), a smooth motion trajectory is planned based on the current servo angle and the final target angle.

[0089] Linear interpolation (trapezoidal velocity curve): Set the maximum angular velocity and maximum angular acceleration The acceleration and deceleration phases are calculated. However, for simple applications, positional linear interpolation can be used directly:

[0090] in The set transition time (e.g., 0.3 seconds). To control the cycle. This method is simple to implement, but may cause sudden speed changes, and needs to be combined with amplitude limiting.

[0091] S-curve interpolation: Using quintic polynomials or sine functions, position, velocity, and acceleration are all continuous, resulting in smoother motion. For example, using a sine curve:

[0092] in t From 0 to Tsmooth Then maintain the target value.

[0093] Low-pass filtering: First-order low-pass filtering is applied to the target angle sequence to suppress high-frequency abrupt changes.

[0094] in ∈[0,1) is a smoothing factor (e.g., 0.8). k This is the frame number. The filtered angle is then sent to the servo.

[0095] (2) Speed ​​and acceleration limits Regardless of the trajectory planning method used, it is essential to ensure that the servo motor's speed does not exceed its physical limits. The desired speed can be calculated in each control cycle.

[0096] like Exceeding the maximum permissible speed Then proceed at maximum speed:

[0097] It can also limit acceleration and prevent rapid acceleration.

[0098] Through the aforementioned calibration, mapping, and smoothing control methods, this embodiment can smoothly and in real-time convert the continuous curvature of a user's gesture into the precise movement of a dexterous hand servo motor, achieving highly realistic gesture reproduction. This method is not only applicable to rock-paper-scissors robots, but can also be extended to various humanoid robots or robotic arm end effectors that require the imitation of fine hand movements, providing a solid technical foundation for natural human-computer interaction.

[0099] Example 2 Based on the above method, this embodiment provides a robot vision-based gesture recognition and mapping system. This system achieves a complete closed loop from gesture image acquisition to robot action execution through a hardware and software collaborative approach. The system as a whole adopts a modular design, with each functional module having a clear division of labor and working closely together to complete the real-time perception, understanding, and reproduction of gestures.

[0100] like Figure 3 As shown, the core components of this system include an image acquisition module, a data processing module, a gesture recognition module, a motion control module, and an interaction management module. The image acquisition module uses a monocular 2D color camera as its core hardware, responsible for capturing the visual information of the user's hand in real time. This camera is deployed at an appropriate location on the robot body or a standalone interactive terminal to ensure clear coverage of the user's hand operation area. The camera outputs continuous RGB image frames to the back-end processing unit, providing raw data input for the entire recognition process.

[0101] The data processing module, running on an embedded processor or dedicated computing unit, is responsible for preprocessing and feature extraction of the raw images. First, the module performs preprocessing operations such as size normalization and pixel value normalization on the acquired image frames to meet the input requirements of subsequent deep learning models. Then, the module calls a pre-trained and embedded lightweight convolutional neural network model to perform forward inference calculations on each frame, outputting the two-dimensional coordinate information of 21 key points on the hand. To improve the stability of the coordinate data, the data processing module also performs temporal filtering on the coordinate sequence of multiple consecutive frames to eliminate coordinate fluctuations caused by hand tremors or detection noise, forming a smooth key point trajectory data stream.

[0102] The gesture recognition module is the intelligent core of this system, responsible for converting key point coordinates into semantically meaningful gesture commands. This module first performs geometric calculations on the received coordinate data, calculating the bending angle joint by joint using the vector dot product method, and summing the angles of multiple joints for each finger to obtain the total bending angle for each finger. Based on this, the module introduces a normalization mechanism, mapping the total bending angle to continuous bending values ​​within the [0,1] interval, thereby eliminating the influence of differences in hand size among different users. To further enhance personalized adaptability, the module has a built-in dynamic calibration function, which guides the user to make a baseline gesture before the interaction begins, automatically collecting and storing the user's personalized bending extreme value parameters for subsequent accurate normalization. The gesture recognition module also includes a gesture semantic judgment unit, which comprehensively analyzes the continuous bending vectors of the five fingers based on a preset logical rule base. A dynamic threshold adjustment mechanism is used during the judgment process, adaptively optimizing the threshold for distinguishing between bent and straight states based on the user's hand characteristics, avoiding misjudgments caused by fixed thresholds.

[0103] The motion control module is responsible for converting the recognized gesture semantics and their continuous curvature parameters into the physical motion of the robot's actuators. This module pre-stores a curvature-servo angle mapping table or function established through calibration. This mapping relationship fully considers factors such as servo model differences, mechanical transmission characteristics, and nonlinear compensation, ensuring that the curvature value accurately corresponds to the actual rotation angle of the servo. During real-time operation, the module quickly calculates the target angle of each finger's corresponding servo based on the input curvature vector for each frame through table lookup and interpolation, and limits the target angle to prevent it from exceeding the mechanical movement range. To achieve smooth and natural movements, the motion control module also integrates trajectory planning functionality. Within each control cycle, it plans a smooth motion trajectory based on the current position and target position, using linear interpolation, S-curve interpolation, or low-pass filtering to generate continuous servo control commands, ensuring continuous changes in speed and acceleration during robot movement and avoiding abrupt jumps or mechanical impacts.

[0104] The interaction management module, acting as the system's scheduling hub, coordinates the working sequence and data flow of various modules, while managing the entire user interaction process. Upon system startup, this module initializes all hardware devices and loads pre-trained deep learning models and calibration parameter libraries. During interaction, the module controls the start and stop of image acquisition, the triggering of gesture recognition, and the issuance of motion commands based on the current interaction state (e.g., standby, calibration, recognition, reproduction). The module also guides user interaction, such as providing prompts via display or voice to complete baseline gesture calibration, or providing real-time feedback on the current gesture state during recognition, enhancing the transparency and user-friendliness of the interaction. For multi-mode interaction scenarios, the interaction management module can switch between different gesture rule libraries or mapping parameters based on user selection, adapting to diverse application needs such as rock-paper-scissors games, gesture teaching, and action imitation.

[0105] The overall system workflow is as follows: When a user enters the interaction area, the image acquisition module continuously captures hand images and transmits them to the data processing module; the data processing module preprocesses each frame of image and runs a key point detection model, outputting the coordinates of 21 key points; the gesture recognition module receives the coordinate data and sequentially performs joint angle calculation, accumulation and normalization, dynamic threshold determination or fuzzy logic determination, finally outputting the gesture semantics and five-dimensional curvature vector of the current frame; the motion control module obtains the target servo angle based on the curvature vector, generates control commands after trajectory planning and smoothing, and sends them to the robot actuator to drive the robot hand to reproduce the user's gesture in real time; the entire process runs in a millisecond-level cycle, forming a real-time closed-loop mapping from user gestures to robot actions. Under the unified scheduling of the interaction management module, the system can seamlessly connect the various stages of gesture calibration, continuous recognition, and action reproduction, providing users with a natural and smooth human-computer interaction experience.

[0106] The above system can execute the gesture recognition method described in Embodiment 1, and has the corresponding functional modules and beneficial effects of the method. For technical details not described in detail in this embodiment, please refer to the gesture recognition method provided in Embodiment 1 of the present invention.

[0107] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; under the concept of the present invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the present invention as described above, which are not provided in detail for the sake of brevity; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A gesture recognition and mapping method based on robot vision, characterized in that, Including the following steps: S1. Acquire user hand images using a monocular 2D color camera, and use a hand key point detection algorithm to obtain a set of two-dimensional coordinates of multiple key feature points of the hand in real time; S2. Based on the two-dimensional coordinate set, the bending angle of each finger joint is calculated, and the bending angles of multiple joints contained in each finger are summed to obtain the total bending angle of the finger. The total bending angle is then mapped to a preset standardized range through linear normalization, and a continuous bending value representing the degree of finger bending is output. S3. Based on the preset gesture logic rules, combine and determine the continuous bending values ​​of the five fingers, and output the corresponding gesture semantics. S4: Through the pre-calibrated mapping relationship, the continuous curvature value is converted into the target control parameter of the robot actuator, driving the robot actuator to smoothly reproduce the user's gesture in real time.

2. The gesture recognition mapping method as described in claim 1, characterized in that, The multiple key feature points of the hand mentioned in step S1 include: Key point 0 on the wrist: the center point of the wrist joint; The key points of the thumb, from 1 to 4, are as follows: the thumb carpal joint, the thumb metacarpophalangeal joint, the thumb interphalangeal joint, and the thumb tip; The key points of the index finger are 5 to 8, in order: the metacarpophalangeal joint of the index finger, the proximal interphalangeal joint of the index finger, the distal interphalangeal joint of the index finger, and the fingertip of the index finger; The key points of the middle finger are 9 to 12, in order: the metacarpophalangeal joint of the middle finger, the proximal interphalangeal joint of the middle finger, the distal interphalangeal joint of the middle finger, and the fingertip of the middle finger; Key points 13 to 16 of the ring finger are, in order: the metacarpophalangeal joint of the ring finger, the proximal interphalangeal joint of the ring finger, the distal interphalangeal joint of the ring finger, and the fingertip of the ring finger; Key points 17 to 20 on the little finger are, in order: the metacarpophalangeal joint of the little finger, the proximal interphalangeal joint of the little finger, the distal interphalangeal joint of the little finger, and the fingertip of the little finger.

3. The gesture recognition mapping method as described in claim 1, characterized in that, The method for calculating the bending angle of each finger joint in step S2 is as follows: For any joint point B and its two adjacent key points A and C, construct a vector starting from B. and The cosine of the included angle is calculated using the vector dot product formula, and then the bending angle of joint B is obtained. θ : ， Where cos θ The calculation results need to be truncated to the interval [-1, 1].

4. The gesture recognition mapping method as described in claim 2, characterized in that, The total bending angle of each finger in step S2 is calculated as follows: Total thumb flexion angle Angle_thumb = θ 1 + θ 2, θ 1 represents the joint bending angle calculated from key points 0, 1, and 2. θ 2 represents the joint bending angle calculated from key points 1, 2, and 3; Total bending angle of index finger Angle_index = θ 5+ θ 6+ θ 7, of which θ 5. θ 6. θ 7 represents the joint bending angles calculated from key points (0,5,6), (5,6,7), and (6,7,8), respectively. Total bending angle of the middle finger Angle_middle = θ 9+ θ 10 + θ 11 ,in θ 9. θ 10 , θ 11 The joint bending angles are calculated from the key points (0,9,10), (9,10,11), and (10,11,12), respectively. Angle_ring = total bending angle of the ring finger θ 13 + θ 14 + θ 15 ,in θ 13 , θ 14 , θ 15 The joint bending angles are calculated from the key points (0,13,14), (13,14,15), and (14,15,16), respectively. Angle_pinky = Total bending angle of little finger θ 17 + θ 18 + θ 19 ,in θ 17 , θ 18 , θ 19 The joint bending angles are calculated from the key points (0,17,18), (17,18,19), and (18,19,20), respectively.

5. The gesture recognition mapping method as described in claim 1, characterized in that, In step S2, the formula for calculating linear normalization is: in Angle The total bending angle of the finger. Angle min and Angle max These are the typical values ​​of the total bending angle under preset fully bent and fully straightened states, respectively; The typical value of the total bending angle for the thumb, Angle min Take 150°, Angle max Take 300°; for the index, middle, ring, and little fingers, Angle min Take 270°, Angle max Alternatively, at the start of the interaction, guide the user to perform two baseline actions: fully extending their palm and fully clenching their fist. Collect and record the total bending angle of each finger at these times, and use these as personalized values ​​for the user. Angle max and Angle min .

6. The gesture recognition mapping method as described in claim 1, characterized in that, In step S3, the gesture logic rules include at least one of the following determination rules: If the continuous bending value of all five fingers is not greater than the threshold T, it is judged as a "stone" gesture. If the continuous bending value of all five fingers is greater than the threshold T, it is determined to be a "cloth" gesture. If only the index and middle fingers have a continuous bending value greater than the threshold T, it is judged as a "scissors" gesture; If only the continuous bending value of the thumb is greater than the threshold T, it is judged as a "thumbs up" gesture.

7. The gesture recognition mapping method as described in claim 6, characterized in that, In step S3, the threshold is a preset static threshold. The conditions of each gesture logic rule are checked in turn. If multiple rules are met at the same time, the final gesture is determined by priority sorting or confidence weighting. If none of them are met, it is determined to be an unknown or transitional gesture.

8. The gesture recognition mapping method as described in claim 6, characterized in that, In step S3, the threshold adopts a dynamic threshold adjustment mechanism; The dynamic threshold adjustment mechanism employs a calibration-based static personalized threshold method: collecting the average value of the bending degree of each finger when the user makes a "stone" gesture. And the average degree of bending of each finger when making the "cloth" gesture. ; Calculate personalized thresholds for each finger ,in Indicates the first i The threshold corresponding to the root finger; Alternatively, the dynamic threshold adjustment mechanism employs an adaptive threshold method based on real-time statistics, including: Sliding window statistics: Maintain a queue of finger flexion values ​​for the most recent N frames, calculate the quantile of each finger flexion in real time, and use the specified quantile as the threshold for determining the flexion / straightening of that finger. Extreme value tracking update: The maximum and minimum bending of each finger are tracked in real time using an exponential moving average, and the midpoint between the two is used as the real-time threshold.

9. The gesture recognition mapping method as described in claim 1, characterized in that, In step S4, the method for establishing the mapping relationship includes: for each finger, selecting multiple representative bending states, recording the corresponding actual servo angles, and forming a calibration point set; establishing a mapping function between bending degree and servo angle through linear interpolation, polynomial fitting, or cubic spline interpolation, and pre-generating a lookup table.

10. A gesture recognition mapping system based on the method of any one of claims 1 to 9, characterized in that, The system includes the following modules: The image acquisition module uses a monocular 2D color camera to capture image frames of the user's hand in real time; The data processing module is used to preprocess the acquired image frames and call the built-in hand key point detection algorithm to output a set of two-dimensional coordinates of multiple key feature points of the hand. The gesture recognition module receives the two-dimensional coordinate set, obtains the bending angle of each finger joint through geometric calculation, accumulates the total bending angle of each finger, and then outputs a continuous bending degree value representing the degree of finger bending after linear normalization. The gesture recognition module has a built-in dynamic threshold adjustment mechanism and gesture logic rule library, which is used to adaptively optimize the judgment threshold according to the user's hand features and combine the continuous bending degree values ​​of the five fingers to judge and output the corresponding gesture semantics. The motion control module has a pre-stored mapping relationship between curvature and servo angle established through calibration. It receives the continuous curvature values ​​and gesture semantics, converts them into target control parameters for the robot actuator through lookup and interpolation operations, and performs trajectory planning and smoothing on the target control parameters to generate continuous control commands. The interaction management module is connected to the image acquisition module, data processing module, gesture recognition module, and motion control module respectively. It is used to coordinate the working sequence and data flow of each module, manage system startup initialization, interaction state switching, user calibration guidance, and rule base and parameter configuration for multi-mode interaction.

Images