Hand pose detection method and system based on vision

Abstract

The application provides a kind of mechanical hand palm synchronous control method and system based on vision human key point detection, belong to robot control and man-machine interaction technical field.The application uses gesture recognition technology to estimate the posture of target arm, captures and extracts key point data.These key point data include the joint position information of arm and hand, which provides necessary input for subsequent control.The application processes these data and converts joint position information into rudder angle information that can drive each joint of mechanical arm.The application uses Kalman filtering algorithm to filter joint angle data, and then transmits filtered rudder angle information to each joint of mechanical arm through serial communication technology, realizes accurate control of mechanical arm, ensures the stability and real-time of data transmission, so that mechanical arm can quickly respond to gesture changes.

Description

Technical Field

[0001] This invention belongs to the field of robot control and human-computer interaction technology, specifically relating to a vision-based method and system for synchronous control of a robotic hand that detects key human body points. Background Technology

[0002] With the continuous advancement of science and technology, robotics is being applied more and more widely in numerous fields such as industry, healthcare, and services, becoming a key force driving the development of various industries. Against this backdrop, human-computer interaction technology, as a bridge for communication and collaboration between robots and humans, directly impacts the level of robot intelligence and application effectiveness, making it a key research area.

[0003] Traditional robotic arms and hands primarily rely on pre-programmed controls or manual remote control. In pre-programmed control, the robotic arm can only perform fixed tasks according to pre-set instructions, lacking adaptability and flexibility to complex and changing environments. For example, in industrial production, the diversity of products and adjustments to production processes can make pre-programmed controls insufficient. While manual remote control can achieve some degree of flexible control, it requires skilled operators, involves complex procedures, and prolonged operation can lead to fatigue, impacting work efficiency and accuracy.

[0004] The rise of human body key point detection technology has brought new opportunities for the control of robotic arms and hands. This technology can accurately identify and locate key points in various parts of the human body, acquiring the body's posture information. Applying it to the control of robotic arms and hands enables more natural and intuitive human-computer interaction. Users only need to make corresponding movements, and the system can quickly parse and convert these changes into control commands for the robotic arm and hand by detecting changes in the human body's key points, allowing the robotic arm and hand to mimic human movements, greatly reducing the operational threshold.

[0005] In the field of medical rehabilitation, robotic arms and hands based on human key point detection can assist medical staff in providing more precise rehabilitation training for patients; in the field of education, they can be used to develop innovative teaching tools and improve teaching effectiveness; in the field of home services, they can help the elderly and disabled to operate smart devices more conveniently and improve their quality of life. Summary of the Invention

[0006] This invention is based on the above background technology and aims to provide a vision-based method and system for synchronous control of a robotic hand that detects key human body points.

[0007] This invention provides a vision-based method for synchronous control of a robotic hand based on human keypoint detection, comprising the following steps: S10, initializing a camera and capturing images after installing the MediaPipe library; S20, estimating the human pose in the image using the BlazePose model, detecting the human pose in the image, obtaining data on human skeletal keypoints, and annotating and displaying the human skeletal keypoints; S30, detecting the ROI image and hand keypoints of the human hand in the image using the palm_detection_SSD model, and then using a regression network to infer the hand keypoint data; S40, extracting the data of human skeletal keypoints and hand keypoints respectively, and calculating the corresponding joint angle data and finger bending angle data using a vector method; S50, converting the joint angle data and finger bending angle data into servo pulse widths or servo angles, and sending the corresponding control signals through serial communication to drive the servos of the robotic hand, thereby realizing vision-based synchronous control of the robotic hand based on human keypoint detection.

[0008] The vision-based human key point detection robotic hand synchronization control method provided by this invention may also have the following features: Step S10 includes the following sub-steps: S11, install the MediaPipe library using the pip command, import the library using the import statement, and configure basic parameters; S12, continuously capture images from the camera using the cv2.VideoCapture(0) method of OpenCV; S13, verify and establish an image capture loop using the cv2.VideoCapture(0).isOpened method to ensure that the system can read image data in real time, wherein the default capture format of OpenCV is the original BGR format image frame; S14, convert the original BGR format image frame to RGB format using the cv2.cvtColor(image,cv2.COLOR_BGR2RGB) method.

[0009] The vision-based human keypoint detection robotic hand synchronization control method provided by this invention may also have the following features: step S20 includes the following sub-steps: S21, using the BlazePose model to estimate the human pose of the preprocessed RGB format image; S22, using the holistic.process(image) method to detect the human pose estimated by the BlazePose model until the human pose is detected, marking the image as writable, and storing the data of the human skeletal keypoints; S23, converting the image from RGB format to BGR format and displaying it using the cv.imshow method; S24, using the mp_drawing.draw_landmarks method to draw the human skeletal keypoints on the image.

[0010] The vision-based human key point detection robotic hand synchronization control method provided by the present invention may also have the following features: In step S21, the BlazePose model includes a posture detector and a posture tracking network. The posture detector identifies human skeletal key points, the posture tracking network predicts the coordinates of human skeletal key points, confirms the existence of the human body in the current frame, and determines the ROI region of the current frame. If the posture tracking network indicates that there is no human body, the posture detector function is re-run in the next frame.

[0011] The vision-based human keypoint detection robotic hand synchronization control method provided by this invention may also have the following features: step S30 includes the following sub-steps: S31, adjusting the preprocessed RGB format image to a uniform resolution and converting it to a TensorFlow-acceptable tensor format; S32, after extracting features from the tensor format image using the palm_detection_SSD model, generating several anchor boxes on feature maps of different scales of the image, and outputting the corresponding classification score tensor and regression result tensor. The classification score tensor represents the probability of a hand existing in each anchor box, and the regression result tensor contains the probability of a hand existing in each anchor box. S33: Select the anchor box with the highest classification score tensor as the candidate region of the target hand, and determine the hand detection box based on its regression result tensor, then calculate the detection box of the entire hand; S34: Based on the four vertices of the hand detection box, crop the hand ROI region, and perform a counterclockwise rotation on the hand ROI region so that the palm is located at the center of the image and the finger direction is parallel to the image edge, to obtain the centered and rotated corrected hand ROI image; S35: Scale the hand ROI image to a uniform pixel size; S36: Input the scaled ROI image into the regression network for inference, and output the hand key point data.

[0012] The vision-based human key point detection robotic hand synchronization control method provided by this invention may also have the following feature: wherein, in step S32, the scale of the anchor frame is set as follows: , Indicates the dimensions of the anchor frame. This indicates the minimum detection scale. Indicates the maximum detection scale. This represents the hierarchical index of the current feature map. The palm_detection_SSD model extracts image features using convolutional layers with different filter sizes and biases, and then adds non-linearity using activation layers. In step S33, the palm detection box is obtained from the target palm candidate region and the decoding formula, which is: , , , In the above formula, Indicates the center coordinates of the candidate region of the target hand. and This indicates the width and height of the candidate area for the target hand. This indicates the center coordinates of the hand detection frame. and This indicates the width and height of the hand detection frame. , , , The four offsets of the anchor frame represented by each anchor point in the regression result tensor are represented by the following: The hand detection box is calculated as follows: Select the palm center joint and the first joint of the middle finger from the palm key points to calculate the direction vector V, and find its rotation angle α with the vertical direction. Translate the center of the palm detection box along the V direction to the palm center position. Then multiply the width and height of the translated palm detection box by empirical values ​​to obtain the preliminary box. Finally, rotate the four corner coordinates of the preliminary box based on the angle α to form the hand detection box.

[0013] The vision-based human key point detection robotic hand synchronization control method provided by this invention may also have the following features: Step S40 includes the following sub-steps: S41, establishing a three-dimensional coordinate system with the person's standing point as the origin, the front of the person as the positive X-axis direction, the right side as the positive Y-axis direction, and the top of the head as the positive Z-axis direction; S42, extracting human skeletal key points and calculating the corresponding joint angle data; S43, after extracting the hand key point data, defining three arrays FINGER_MCP_INDEX, FINGER_PIP_INDEX, and FINGER_DIP_INDEX to store the proximal phalanx key points, distal phalanx key points, and distal phalanx key points of the fingers, respectively; S44, based on the data of FINGER_MCP_INDEX, FINGER_PIP_INDEX, and FINGER_DIP_INDEX, traversing the five fingers to calculate the finger bending angle data of each finger.

[0014] The vision-based human key point detection robotic hand synchronization control method provided by this invention may also have the following features: In step S42, the holistic.process(image).pose_landmarks method is used to check whether the BlazePose model has detected human skeletal key points. If detected, the data of the human skeletal key points is stored in the RHL variable, and then the list pose_angle is initialized to store the calculated joint angle data; In step S43, the holistic.process(image).right_hand_landmarks method is used to check whether hand key points have been detected and obtained. If detected, the data of the hand key points is stored in the RHL variable, and then the list hand_angle is initialized to store the calculated finger bending angle data.

[0015] The vision-based human key point detection robotic hand synchronization control method provided by this invention may also have the following feature: In step S50, Kalman filtering is further used to smooth the joint angle data and finger bending angle data. The mapping method between the joint angle data and the servo motor pulse width is as follows: , , In the above formula, , , Joint angle data after Kalman filtering and smoothing , , Corresponding values, joint angle data , , These represent the rotation angles of the right shoulder in the YZ plane, the right shoulder in the XZ plane, and the right elbow, respectively. This indicates the pulse width of the servo motor corresponding to the rotation of the right shoulder in the YZ plane. This represents the pulse width of the servo motor corresponding to the rotation of the right shoulder in the XZ plane. The pulse width of the servo motor corresponding to the right elbow is represented by the following formula: (This formula is used to convert finger flexion angle data to servo motor angle.) , The above two formulas are the angle conversion formulas for the thumb and four fingers of the robotic hand, respectively. and To correspond to the maximum and minimum amplitude of the servo motor movement, These are the corresponding values ​​of the finger bending angle data after Kalman filtering smoothing. For output The minimum value in, , For output The maximum value in and The difference, or This corresponds to the difference between the maximum and minimum amplitude of the servo motor movement.

[0016] This invention also provides a vision-based human keypoint detection robotic hand synchronization control system, characterized by using any of the aforementioned vision-based human keypoint detection robotic hand synchronization control methods, comprising: an image acquisition module for real-time capture of images containing a human arm and hand via a camera; a keypoint detection module integrating a MediaPipe library for obtaining human skeletal keypoints and hand keypoints and corresponding data from the images; a coordinate processing module for extracting the data of human skeletal keypoints and hand keypoints respectively, and calculating the corresponding joint angle data and finger bending angle data; a Kalman filter module for smoothing the output data of the coordinate processing module; a control signal conversion module for converting the smoothed joint angle data and finger bending angle data into servo pulse widths and sending corresponding control signals via serial communication; and a robotic hand module for receiving control signals to achieve vision-based human keypoint detection robotic hand synchronization control.

[0017] The role and effect of the invention:

[0018] Based on the needs of human-computer interaction, this invention comprehensively utilizes advanced image processing technology, machine learning algorithms, filtering technology, and communication technology to achieve precise control of the robotic arm.

[0019] In the implementation of the control system, this invention first utilizes gesture recognition technology to estimate the posture of the target arm, capturing and extracting key point data. This key point data includes the joint position information of the arm and hand, providing the necessary input for subsequent control. Subsequently, the system processes this data through a series of algorithms, converting the joint position information into servo motor rotation angle information that can drive the various joints of the robotic arm.

[0020] To ensure the stability and real-time performance of the output, this invention employs a Kalman filter algorithm to filter the joint angle data. The Kalman filter can effectively estimate the true state of the system from noisy measurements, reducing random fluctuations caused by sensor noise and other interference factors, resulting in a smoother and more reliable system output.

[0021] This invention utilizes serial communication technology to transmit filtered servo motor angle information to various joints of the robotic arm, achieving precise control of the robotic arm. Serial communication technology ensures the stability and real-time performance of data transmission, enabling the robotic arm to quickly respond to changes in human gestures.

[0022] The entire process of this invention involves a series of complex operations, from data acquisition and processing to decision-making and execution. The design of this invention fully considers the requirements of real-time performance, accuracy, and stability, demonstrating excellent performance through actual arm-following and hand-following experiments. The robotic arm can capture and recognize human postures in real time, accurately simulating continuous arm postures and specific hand gestures, and the system exhibits fast response and good stability. Attached Figure Description

[0023] Figure 1 This is a flowchart of a vision-based human key point detection robotic hand synchronization control method according to an embodiment of the present invention.

[0024] Figure 2 This is a flowchart of steps S10-S20 of an embodiment of the present invention for image capture and human skeletal key point detection.

[0025] Figure 3 This is the output display diagram after marking and drawing the key points of the human skeleton in step S20 of an embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram of the input and output of the palm_detection_SSD model in step S30 of an embodiment of the present invention.

[0027] Figure 5 This is the output display diagram of the palm_detection_SSD model in step S33 of an embodiment of the present invention.

[0028] Figure 6 This is the output display diagram of the palm_detection_SSD model after regression network diagnosis in step S36 of an embodiment of the present invention.

[0029] Figure 7 This is a flowchart of the right arm key point data processing procedure in step S42 of an embodiment of the present invention.

[0030] Figure 8 These are the annotations of key points of the human skeleton and key points of the palm in the embodiments of the present invention.

[0031] Figure 9 This is a flowchart of the right-hand key point data processing in step S44 of an embodiment of the present invention.

[0032] Figure 10 This is a schematic diagram of the bending angle data of a joint after Kalman filtering in step S51 of an embodiment of the present invention.

[0033] Figure 11This is a schematic diagram of the connection relationship of a vision-based human key point detection robotic hand synchronous control system according to an embodiment of the present invention. Detailed Implementation

[0034] To make the technical means, creative features, objectives and effects of this invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate a vision-based human key point detection robotic hand synchronous control method and system of this invention.

[0035] <Example>

[0036] Figure 1 This is a flowchart of a vision-based human key point detection robotic hand synchronization control method according to an embodiment of the present invention.

[0037] like Figure 1 As shown, this embodiment provides a vision-based method for synchronous control of a robotic hand that detects key human body points, including the following steps:

[0038] S10, After installing the MediaPipe library, initialize the camera and capture images, including the following sub-steps S11~S14:

[0039] S11. Use the pip command to install the MediaPipe library, import the library using the import statement, and configure the basic parameters.

[0040] The MediaPipe library is a powerful toolset that integrates a variety of machine learning vision algorithms, specifically designed for performing gesture recognition and pose estimation tasks.

[0041] S12, continuously capture images from the camera using OpenCV's cv2.VideoCapture(0) method.

[0042] S13, the cv2.VideoCapture(0).isOpened method is used to verify and establish an image capture loop to ensure that the system can read image data in real time. The default capture format of OpenCV is the original BGR format image frame.

[0043] S14, use the cv2.cvtColor(image,cv2.COLOR_BGR2RGB) method to convert the original BGR format image frame to RGB format.

[0044] S20, Human Skeletal Keypoint Detection: After estimating the human pose in the image using the BlazePose model, the human pose in the image is detected, the human skeletal keypoint data is obtained, and the human skeletal keypoints are annotated and displayed on them, including the following sub-steps S21~S24:

[0045] S21. Use the BlazePose model to estimate human pose in a preprocessed RGB format image.

[0046] The BlazePose model is a lightweight convolutional neural network architecture optimized for real-time inference on mobile devices for human pose estimation. It consists of a lightweight pose detector and a pose tracking network.

[0047] The system consists of a pose detector that identifies key points on the human skeleton; a pose tracking network that predicts key point coordinates, confirms the presence of the human body in the current frame, and determines the Region of Interest (ROI) of the current frame. When the pose tracking network indicates that no one is present, the system will re-run the pose detector function in the next frame to continue pose estimation.

[0048] S22, in the holistic module of MediaPipe, the holistic.process(image) method is used to detect the human pose estimated by the BlazePose model until the human pose is detected. After that, the image is marked as writable and the data of human skeletal key points are stored.

[0049] S23. After converting the image from RGB format back to BGR format, it is displayed in the OpenCV environment using the cv.imshow method, thus visually demonstrating the pose estimation results and providing a visual reference for further analysis and processing. Specifically, as shown below... Figure 3 As shown.

[0050] S24, use the mp_drawing.draw_landmarks method to draw the key points of the human skeleton on the image.

[0051] S30, Hand detection and hand keypoint detection: After detecting the ROI image of the human hand and hand keypoints in the image using the palm_detection_SSD model, a regression network is used to infer the hand keypoint data, specifically including the following sub-steps S31~S36:

[0052] S31 adjusts RGB format images to a uniform resolution of 128×128 and converts them to a tensor format acceptable to TensorFlow.

[0053] Specifically, it is a four-bit tensor with dimensions of 1×128×128×3, where 1 represents the batch size, 128×128 is the image resolution, and 3 corresponds to the three RGB color channels, such as... Figure 4 As shown in part (a) of the document.

[0054] S32, process the tensor format image to obtain the classification score tensor and the regression result tensor, including the following sub-steps S32-1~S32-3:

[0055] S32-1 uses the palm_detection_SSD model to extract features from tensor format images.

[0056] In the convolutional layer portion of the palm_detection_SSD model, the input tensor is processed through multiple convolutional layers. These layers utilize filters of different sizes and biases to extract image features. An activation layer typically follows the convolutional operation, adding non-linearity to the output of the convolutional layers and enhancing the model's expressive power.

[0057] S32-2, as shown Figure 4 As shown in section (b), the palm_detection_SSD model generates 896 anchor boxes on feature maps of different scales in the image to predict the position of the palm. The size and number of channels of these feature maps are clearly labeled in the figure; for example, 1×16×16×128 indicates that there are 128 feature channels per grid point on a 16×16 grid.

[0058] The dimensions of the anchor frame are set as follows:

[0059] .

[0060] In the above formula, Indicates the dimensions of the anchor frame. This indicates the minimum detection scale. Indicates the maximum detection scale. This indicates the hierarchical index of the current feature map.

[0061] According to the above formula, these anchor boxes are defined on feature maps at two different scales: one set of anchor boxes is generated on a 16×16 resolution feature map using two different scales (0.12 and 0.23, respectively); the other set of anchor boxes is generated on an 8×8 resolution feature map using six different scales (0.35, 0.45, 0.61, 0.74, 0.87 and 1, respectively).

[0062] S32-3, output the corresponding classification score tensor and regression result tensor:

[0063] After the palm_detection_SSD model completes inference, it will output two tensors:

[0064] A 1×896×1 classification score tensor contains classification scores for 896 anchor points. These scores range from 0 to 1, reflecting the probability of a hand being detected at each anchor point (the closer the value is to 1, the greater the likelihood of detecting a hand). In the experiment, a threshold of 0.60 was set for the classification score; if the score is below this threshold, it is considered that no hand was detected at the corresponding anchor point.

[0065] The other is a 1×896×18 regression result tensor, containing the regression result for each anchor point, where each anchor point has 4 values ​​representing the offset of the bounding box. , , , ) and the coordinates of 7 key points on the palm (each key point has ) and (Two coordinates), totaling 18 values. These offsets and keypoint coordinates help to accurately locate the position of the hand.

[0066] After the palm_detection_SSD model completes inference and generates classification and regression result tensors, the next step is to decode these results to determine the precise location of the hand:

[0067] S33, Select the anchor box with the highest classification score tensor as the candidate region of the target hand, and determine the hand detection box based on its regression result tensor, then calculate the detection box of the entire hand, including the following sub-steps S33-1~S33-2:

[0068] S33-1, Determine the palm detection bounding box palm_rect:

[0069] Since this embodiment is based on detecting only one hand in the image, the decoding process selects the anchor box with the highest classification score from 896 anchor boxes as the candidate region for the target hand. The center coordinates of this candidate region are marked as follows: Width and height are respectively denoted as and The actual final detection result shows that the center coordinates of the palm detection frame are as follows: Width and height are respectively denoted as and The calculation formula (decoding formula) is as follows:

[0070] .

[0071] .

[0072] .

[0073] .

[0074] After the above decoding formula, and based on the classification and regression results provided by the palm_detection_SSD model, the palm detection box palm_rect is obtained, whose center coordinates are determined by... The given value is provided, and its width and height are respectively... and In this step, the palm detection bounding box `palm_rect` is as follows: Figure 5 As shown in the blue box, there are seven white keypoints, which are the seven palm keypoints output by the palm_detection_SSD model in step S32-3.

[0075] S33-2, Calculate the entire hand detection bounding box hand_rect:

[0076] First, select the center joint of the palm and the first joint of the middle finger from the palm detection bounding box palm_rect, and calculate the vector V using the coordinates of these two points. Figure 5 (Represented by yellow lines in the middle), and calculate the angle between the vector and the y-axis (vertical direction) to determine the rotation angle α of the palm.

[0077] Then, the center coordinates of the palm detection box palm_rect are translated along the direction of vector V to the approximate position of the palm.

[0078] Next, multiply the width and height of the resulting rectangle by an empirical factor (usually 2.8) to obtain... Figure 5 The orange box in the image. Then, based on the calculated rotation angle α, a coordinate transformation is performed on the coordinates of the four corners of the orange box to form... Figure 5 The red box in the image represents the entire hand detection bounding box, hand_rect. It includes not only the palm but also all the fingers, ensuring that all important hand features are covered in subsequent keypoint detection.

[0079] S34. Based on the four vertices of the hand detection bounding box hand_rect, the hand ROI region is cropped out. The hand ROI region is rotated counterclockwise by an angle of α so that the palm is located at the center of the image and the finger direction is parallel to the edge of the image, thus obtaining the centered and rotated corrected hand ROI image.

[0080] S35, scale the hand ROI image (i.e., the red box around the hand) to a uniform size of 224×224 pixels.

[0081] S36. The scaled ROI image is input into the regression network for inference, and the hand key point data is output. A total of three results are output:

[0082] (1) A 1×1 classification result gives a score, which indicates the confidence level of the detected key point, i.e. the reliability of the key point.

[0083] (2) The 1×1 classification result is used to distinguish whether the detected hand is the left or right hand. When the output value is close to 1, it means that the detected hand is more likely to be the right hand; if the value is close to 0, it means that it is more likely to be the left hand.

[0084] (3) A 1×63 regression result, which contains the three-dimensional coordinate information of 21 skeletal points in the current detection image.

[0085] like Figure 6 As shown, these coordinates include values ​​in the X, Y, and Z directions. The X and Y coordinates are normalized and range from 0 to 1, representing the position of the skeletal point in the original hand image. The Z coordinate provides the distance of the skeletal point relative to keypoint 0, the lowest point of the hand closest to the wrist (e.g., ...). Figure 6 part a in Figure 8 As shown in part b), its value range is between -0.5 and 0.5. The Z coordinate of key point 0 is always set to 0.0 as a reference point, and the Z coordinates of the other key points indicate whether they are closer (positive value) or farther (negative value) from the camera relative to point 0. Thus, all the key point 3D data of the hand are obtained.

[0086] S40, after extracting the data of key points of the human skeleton and key points of the hand, calculate the corresponding joint angle data and finger bending angle data using the vector method, including the following sub-steps S41~S44:

[0087] S41. Establish a three-dimensional coordinate system with the person's standing point as the origin, the front of the person as the positive X-axis, the right side as the positive Y-axis, and the top of the head as the positive Z-axis.

[0088] S42, as Figure 7 As shown, extracting key points of the human skeleton and calculating the corresponding joint angle data includes the following sub-steps S42-1 to S42-3:

[0089] S42-1 checks whether human skeletal keypoints are detected using the holistic.process(image).pose_landmarks method. If detected, these human skeletal keypoints are stored in the RHL variable.

[0090] S42-2, Calculate joint angle data:

[0091] (1) Calculate the joint angle data of the right shoulder in the YZ plane.

[0092] Three key points were extracted from RHL, namely: Figure 8 Key points 11, 12, and 14 of the human skeleton in section a correspond to the spatial coordinates of the left shoulder, right shoulder, and right elbow. Define a function "calculate_pose_angle2dyz(landmark1,landmark2,landmark3)" to calculate the included angle using the spatial coordinates of the three points.

[0093] To calculate the joint angles of the right shoulder in the YZ plane, given the spatial coordinates of three points, two vectors can be obtained. The vector formula can then be used.

[0094] .

[0095] in, It is a point Time The vector, and These are the three-dimensional coordinates of these two points. Let A, B, C, and D be the three key points, 11, 12, and 14 respectively. B C A point can yield two vectors:

[0096] .

[0097] .

[0098] Since the projection is onto the YZ plane, the X coordinate is 0. Therefore, the vector is:

[0099] .

[0100] .

[0101] After obtaining the vectors, use the dot product formula to calculate the angle θ between the two vectors:

[0102] (Equation 1).

[0103] in, and Dot product of two vectors for:

[0104] (Equation 2).

[0105] vector and The moduli are respectively:

[0106] (Equation 3).

[0107] (Equation 4).

[0108] Substituting equations 1, 2, 3, and 4, we get and The included angle The cosine value is:

[0109] .

[0110] but and The included angle The value is:

[0111] .

[0112] (2) Calculate the joint angle data of the right shoulder in the XZ plane.

[0113] That is, to project onto the XZ plane, the Y coordinate is 0. Using the aforementioned... The vector formula yields the vector:

[0114] .

[0115] .

[0116] Substituting equations 1, 2, 3, and 4, we obtain... and The included angle The absolute value of the cosine is:

[0117] .

[0118] but and The included angle The value is:

[0119] .

[0120] Meanwhile, since the range of motion of the human arm in the XZ plane is 0°~180°, the servo motor controlling the movement of the right shoulder in the XZ plane is limited accordingly. If it exceeds 180°, the corresponding calculation is performed based on 180°. The remaining calculation process is similar to the movement of the right shoulder in the YZ plane, and finally the joint angle data is obtained.

[0121] (3) Calculate the joint angle data of the right elbow movement.

[0122] Three key points were extracted from RHL, namely: Figure 8Key points 12, 14, and 16 of the human skeleton in section a correspond to the spatial coordinates of the right shoulder, right elbow, and right wrist. The calculation method is similar to that described above. Let key points 12, 14, and 16 be... , , Point, from the aforementioned The vector formula yields two vectors:

[0123] .

[0124] .

[0125] Substituting equations 1, 2, 3, and 4, we obtain... and The included angle The cosine value is:

[0126] .

[0127] but and The included angle The value is:

[0128] .

[0129] This gives us the joint angle data for the entire arm.

[0130] S42-3, Initialize the list pose_angle, which is used to store the joint angle data for subsequent calculations.

[0131] S43, after extracting the key point data of the hand, define three arrays: FINGER_MCP_INDEX, FINGER_PIP_INDEX, and FINGER_DIP_INDEX, to store the key points of the proximal phalanges, the distal phalanges, and the distal phalanges of the fingers, respectively, including the following sub-steps S43-1~S43-2:

[0132] In step S43-1, the `holistic.process(image).right_hand_landmarks` method is used to check if right-hand keypoints are detected in the hand keypoints. If detected, these keypoints are stored in the `RHL` variable. Next, the list `hand_angle` is initialized to store the calculated finger bending angle data.

[0133] S43-2, define three arrays FINGER_MCP_INDEX, FINGER_PIP_INDEX, and FINGER_DIP_INDEX, respectively storing... Figure 8The following key points are in part b: (1) Metacarpophalangeal joint (MCP) key points, namely key points 1, 5, 9, 13 and 17; (2) Proximal interphalangeal joint (PIP) key points, namely key points 2, 6, 10, 14 and 18; (3) Distal interphalangeal joint (DIP) key points, namely key points 3, 7, 11, 15 and 19.

[0134] S44. Based on the data of FINGER_MCP_INDEX, FINGER_PIP_INDEX and FINGER_DIP_INDEX, iterate through the five fingers to calculate the finger bending angle data of each finger.

[0135] When calculating the degree of finger flexion, the coordinate changes of the three key joints—MCP, PIP, and DIP—are directly related to the finger's flexion movement, providing crucial information for precise gesture capture and robotic arm control. Therefore, the degree of flexion of each finger can be calculated based on the angles they form, as shown in the calculation method below. Figure 9 As shown:

[0136] The degree of thumb flexion is calculated, and a loop is set to iterate through all five fingers. Three proximal, distal, and terminal phalanx keypoints of the thumb are extracted from the RHL. Figure 8 Key points 1, 2, and 3 in part b. Define a function "calculate_finger_angle(MCP,PIP,DIP)" to calculate the angle between the three points using their spatial coordinates.

[0137] set up Figure 8 The three key points 1, 2, and 3 in part b are respectively , , Point, from the aforementioned The vector formula yields two vectors:

[0138] .

[0139] .

[0140] Substituting equations 1, 2, 3, and 4, we obtain... and The included angle The cosine value is:

[0141] .

[0142] Then the included angle can be obtained. The value represents the degree of bending of the thumb. Similarly, the bending angle data of the remaining four fingers can be obtained by iterating through the loop.

[0143] S50 uses Kalman filtering to smooth the joint angle data and finger bending angle data, then converts the corresponding data into servo motor pulse widths and sends the corresponding control signals through serial communication to drive the servo motors of the robotic hand, thereby realizing vision-based human key point detection and synchronous control of the robotic hand. This includes the following sub-steps:

[0144] S51 uses Kalman filtering to smooth the joint angle data and finger bending angle data:

[0145] The palm_detection_SSD model can be affected by various noises when capturing hand keypoints, such as camera noise, hand shakiness, and ambient light variations. These noises can lead to uneven data acquisition and may even contain outliers, directly impacting the accuracy of the gesture-to-servo control signal conversion. This step uses a Kalman filter algorithm to optimize the response accuracy.

[0146] The Kalman filter is a state estimation algorithm for linear dynamic systems that effectively estimates the true state of a system from noisy measurements. This is achieved through two main steps: prediction and update. First, prediction uses the system's dynamic model to forecast the state at the next moment. In the `palm_detection_SSD` model, this can predict the location of key gesture points in the next frame. Second, update occurs when new measurement data arrives. The Kalman filter compares the predicted state with the actual observed data and adjusts the prediction based on the differences to obtain a more accurate estimate.

[0147] This step creates two Kalman filter instances, specifying a state vector dimension of 4 and an observation vector dimension of 4, respectively, to filter the servo angle data of the robotic arm and the robotic hand, in order to reduce noise and improve control accuracy.

[0148] Since the human arm movement is designed to be slow, with the hand moving at an approximately uniform linear speed in front of the camera, it can be considered a linear system. The prediction phase begins with estimating the state at the current moment. It is calculated based on the state estimate and process model from the previous time step. The state vector typically contains all the internal states of the system, such as the gesture angles of all fingers, and the process model can be represented as a state transition matrix. It describes how the state evolves over time. Therefore, the predicted state can be represented as:

[0149] .

[0150] in, Indicates the current moment.

[0151] This step assumes that the angle information does not change during the prediction process, that is, the rate of change of the angle information during the state transition is 0. Based on this, the state transition matrix kf.F is set as follows:

[0152] kf.F=np.array([[1,0,0,0,0]

[0153] [0,1,0,0,0]

[0154] [0,0,1,0,0]

[0155] [0,0,0,1,0]

[0156] [0,0,0,0,1]])

[0157] As shown above, the Kalman filter state transition matrix kf.F describes the relationship between the current state and the state at the next time step.

[0158] Next, we set up a measurement matrix kf.H, which describes the relationship between state variables and observed variables. Here, we assume that angle information can be directly observed, meaning that the angle information in the measurement matrix is ​​directly mapped to the state vector:

[0159] kf.H=np.array([[1,0,0,0,0]

[0160] [0,1,0,0,0]

[0161] [0,0,1,0,0]

[0162] [0,0,0,1,0]

[0163] [0,0,0,0,1]])

[0164] At the same time, the Kalman filter also predicts the error covariance matrix. It describes the uncertainty of the predicted state. Error covariance matrix. The update formula is:

[0165] .

[0166] in, It is the process noise covariance matrix, representing the uncertainty of the system during its time evolution. express The transpose of .

[0167] Kalman filters utilize new measurement data To correct the prediction. Measurement model. This describes how the state vector is mapped to the measurement vector. The measurement vector may be obtained directly from the sensor, such as the observed coordinates of key points on the hand.

[0168] Measurement updates involve two main steps: calculating the Kalman gain. And update the state estimate.

[0169] Kalman gain is the measurement noise covariance matrix. and prediction error covariance matrix The function is calculated using the following formula:

[0170] .

[0171] The Kalman gain determines the degree to which the measurement data influences the state estimate. Then, the Kalman gain is used to update the state estimate:

[0172] .

[0173] in, This is the corrected state estimate, which combines the predicted information with the new measurement data. Finally, the error covariance matrix is ​​updated:

[0174] .

[0175] In the above formula, Represents the identity matrix.

[0176] The update of the error covariance matrix reflects how new measurement data reduces the uncertainty of prediction.

[0177] In this step, it is assumed that the process noise is constant and independent. During the update phase, the process noise covariance matrix is ​​set. and measurement noise covariance matrix The parameters are 0.01 and 0.1 respectively. The initial state vector and initial covariance matrix are set to np.array([0, 0, 0, 0, 0]) and 10 respectively. The second Kalman filter is set in the same way.

[0178] During the filtering process, the "kf.predict" method is used to predict the state at the next time step, the "kf.update" method is used to update the current state, the observed values ​​are used for correction, and finally the "kf.x" method is used to output the filtered value. Throughout the process, the Kalman filter iterates between prediction and update to provide a continuous estimate of the system state. As long as the system continues to repeat the above two steps, the system state will continuously follow the observed values ​​and have a certain filtering effect. This recursive characteristic makes the Kalman filter very suitable for gesture recognition and robotic arm control systems, where the data stream is continuous and real-time.

[0179] Figure 10 This is a schematic diagram of the bending angle data of a joint after Kalman filtering in step S51 of an embodiment of the present invention. Figure 10 As shown in the figure, the jitter in the joint angle data obtained after the above steps has been significantly improved.

[0180] Kalman filters reduce random fluctuations caused by sensor noise and other interference factors through their mathematical model, and the output data is smoother, helping to avoid malfunctions caused by sudden data changes. Furthermore, Kalman filters provide stability analysis, helping the system remain stable in the face of uncertainty. Suitable for real-time systems, they have fixed computational complexity and can be executed quickly, which is crucial for real-time gesture recognition and robotic arm control. By using Kalman filters, gesture recognition systems can provide more accurate and reliable hand posture information.

[0181] S52 converts the corresponding data into servo pulse width or servo control signal:

[0182] (1) Use analytical methods to convert the joint angle data obtained in step S42-2 into the corresponding pulse width of the servo rotation angle, and design the corresponding formula based on the correspondence between the PWM signal and the servo angle:

[0183] .

[0184] .

[0185] .

[0186] In the above formula, , , After Kalman filtering smoothing in step S51 , , data, This indicates the pulse width of the servo motor corresponding to the rotation of the right shoulder in the YZ plane. This represents the pulse width of the servo motor corresponding to the rotation of the right shoulder in the XZ plane. This indicates the pulse width of the servo motor corresponding to the right elbow.

[0187] (2) The finger bending angle data obtained in step S44 is normalized to convert it into a servo control signal. The conversion formula between this angle and the servo angle is obtained from the experiment:

[0188] .

[0189] .

[0190] The above two formulas are the angle conversion formulas for the thumb and four fingers of the robotic hand, respectively. Among them, and To correspond to the maximum and minimum amplitude of the servo motor movement, The finger bending angle data after Kalman filtering smoothing in step S51 Corresponding value, For output The minimum value in, , For output The maximum value in and The difference, or This corresponds to the difference between the maximum and minimum amplitude of the servo motor movement.

[0191] S53 sends corresponding control signals through serial communication to drive the servo motor of the robotic arm, thereby realizing vision-based human key point detection and synchronous control of the robotic arm.

[0192] Figure 11 This is a schematic diagram of the connection relationship of a vision-based human key point detection robotic hand synchronous control system according to an embodiment of the present invention.

[0193] like Figure 11 As shown, this embodiment also provides a vision-based human key point detection robotic hand synchronization control system, which uses a vision-based human key point detection robotic hand synchronization control method provided in this embodiment, including an image acquisition module 10, a key point detection module 20, a coordinate processing module 30, a Kalman filter module 40, a control signal conversion module 50, and a robotic hand module 60.

[0194] The image acquisition module 10 is used to capture images containing human arms and hands in real time via a camera.

[0195] The key point detection module 20 integrates the MediaPipe library, which is used to obtain the human skeleton key points and hand key points and corresponding data of the human body according to the image acquired by the image acquisition module 10 according to the method of steps S10~S30.

[0196] The coordinate processing module 30 is used to extract the data of key points of the human skeleton and key points of the hand according to the method in step S40, and then calculate the corresponding joint angle data and finger bending angle data.

[0197] The Kalman filter module 40 is used to smooth the output data of the coordinate processing module 30.

[0198] The control signal conversion module 50 is used to convert the smoothed joint angle data and finger bending angle data into servo pulse width according to the method in step S50 and send the corresponding control signal through serial communication.

[0199] The robotic arm module 60 is used to receive control signals sent by the control signal conversion module 50, thereby realizing vision-based human key point detection and synchronous control of the robotic arm.

[0200] Those skilled in the art should understand that this invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to this invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A vision-based method for synchronous control of a robotic hand that detects key human body points, characterized in that, Includes the following steps: S10, after installing the MediaPipe library, initializes the camera and captures images; S20, after estimating the human pose in the image using the BlazePose model, the human pose in the image is detected, the data of the human skeleton key points are obtained, and the human skeleton key points are annotated, drawn, and displayed on them, including the following sub-steps: S21, use the BlazePose model to estimate the human pose of the preprocessed RGB format image. S22, the holistic.process(image) method is used to detect the human pose estimated by the BlazePose model until a human pose is detected. After that, the image is marked as writable, and the data of the human skeleton key points are stored. S23, after converting the image from RGB format to BGR format, display it using the cv.imshow method. S24, Use the mp_drawing.draw_landmarks method to draw the key points of the human skeleton on the image; S30, After using the palm_detection_SSD model to detect the ROI image and key points of the human hand in the image, a regression network is used to infer the data of the key points of the hand; S40, after extracting the data of the key points of the human skeleton and the key points of the hand respectively, calculate the corresponding joint angle data and finger bending angle data by vector method; S50, the joint angle data and the finger bending angle data are converted into servo motor pulse width or servo motor angle and the corresponding control signals are sent through serial communication to drive the servo motor of the robotic hand, thereby realizing vision-based human key point detection and synchronous control of the robotic hand.

2. The vision-based human key point detection robotic hand synchronization control method according to claim 1, characterized in that: in, Step S10 includes the following sub-steps: S11. Use the pip command to install the MediaPipe library, import the library using the import statement, and configure the basic parameters. S12, continuously capture images from the camera using OpenCV's cv2.VideoCapture(0) method; S13, the cv2.VideoCapture(0).isOpened method is used to verify and establish an image capture loop to ensure that the system can read image data in real time. The default capture format of OpenCV is the original BGR format image frame. S14, use the cv2.cvtColor(image,cv2.COLOR_BGR2RGB) method to convert the original BGR format image frame to RGB format.

3. The vision-based human key point detection robotic hand synchronous control method according to claim 1, characterized in that: in, In step S21, the BlazePose model includes a pose detector and a pose tracking network. The posture detector identifies key points of the human skeleton. The posture tracking network predicts the coordinates of key points on the human skeleton, confirms the presence of the human body in the current frame, and determines the ROI region of the current frame. If the posture tracking network indicates that there is no human body, the posture detector function is re-run in the next frame.

4. The vision-based human key point detection robotic hand synchronization control method according to claim 1, Its features are: in, Step S30 includes the following sub-steps: S31, adjust the preprocessed RGB format image to a uniform resolution and convert it to a tensor format acceptable to TensorFlow; S32, after extracting features from the image in tensor format using the palm_detection_SSD model, several anchor boxes are generated on the feature maps of the image at different scales, and the corresponding classification score tensor and regression result tensor are output. The classification score tensor represents the probability that each anchor box contains a hand, and the regression result tensor contains the bounding box offset of each anchor box and the coordinates of several hand key points. S33, select the anchor box with the highest classification score tensor as the candidate region of the target hand, and determine the hand detection box based on its regression result tensor, and then calculate the detection box of the whole hand. S34. Based on the four vertices of the hand detection box, the hand ROI region is cropped out, and the hand ROI region is rotated counterclockwise so that the palm is located in the center of the image and the finger direction is parallel to the edge of the image, thus obtaining the centered and rotated corrected hand ROI image. S35, scale the hand ROI image to a uniform pixel size; S36. The scaled ROI image is input into the regression network for inference, and the data of the key points of the hand are output.

5. The vision-based human key point detection robotic hand synchronization control method according to claim 4, characterized in that: in, In step S32, the dimensions of the anchor frame are set as follows: ， Indicates the dimensions of the anchor frame. This indicates the minimum detection scale. Indicates the maximum detection scale. This indicates the hierarchical index of the current feature map. The palm_detection_SSD model extracts features from the image using convolutional layers with different sized filters and biases, and then adds non-linearity using activation layers. In step S33, the palm detection box is obtained from the target palm candidate region and the decoding formula, wherein the decoding formula is: ， ， ， ， In the above formula, This indicates the center coordinates of the candidate target hand area. and This indicates the width and height of the target palm candidate region. This indicates the center coordinates of the hand detection frame. and This indicates the width and height of the hand detection frame. , , , These represent the four offsets of the anchor frame represented by each anchor point in the regression result tensor; The hand detection frame is calculated as follows: the center joint of the palm and the first joint of the middle finger are selected from the key points of the palm to calculate the direction vector V, and the rotation angle α between the vector and the vertical direction is calculated. The center of the hand detection frame is translated along the V direction to the center of the palm. Then, the width and height of the translated hand detection frame are multiplied by empirical values ​​to obtain a preliminary frame. Finally, the four corner coordinates of the preliminary frame are rotated based on the angle α to form the hand detection frame.

6. The vision-based human key point detection robotic hand synchronization control method according to claim 1, Its features are: in, Step S40 includes the following sub-steps: S41. Establish a three-dimensional coordinate system with the person's standing point as the origin, the front of the person as the positive X-axis, the right side as the positive Y-axis, and the top of the head as the positive Z-axis. S42, extract the key points of the human skeleton and calculate the corresponding joint angle data; S43, after extracting the data of the key points of the hand, define three arrays FINGER_MCP_INDEX, FINGER_PIP_INDEX and FINGER_DIP_INDEX to store the key points of the proximal phalanges of the fingers, the key points of the distal phalanges of the fingers and the key points of the distal phalanges of the fingers, respectively. S44. Based on the data of FINGER_MCP_INDEX, FINGER_PIP_INDEX and FINGER_DIP_INDEX, iterate through the five fingers to calculate the finger bending angle data of each finger.

7. The vision-based human key point detection robotic hand synchronization control method according to claim 6, characterized in that: in, In step S42, the holistic.process(image).pose_landmarks method is used to check whether the BlazePose model has detected the human skeleton key points. If it has, the data of the human skeleton key points is stored in the RHL variable. Then, the pose_angle list is initialized to store the calculated joint angle data. In step S43, the holistic.process(image).right_hand_landmarks method is used to check whether the hand key points have been detected and obtained. If detected, the data of the hand key points is stored in the RHL variable. Then, the list hand_angle is initialized to store the calculated finger bending angle data.

8. The vision-based human key point detection robotic hand synchronization control method according to claim 6, characterized in that: in, In step S50, Kalman filtering is also used to smooth the joint angle data and the finger bending angle data. The mapping method between the joint angle data and the servo pulse width is as follows: ， ， ， In the above formula, , , The joint angle data after being smoothed by the Kalman filter. , , Corresponding value, the joint angle data , , These represent the rotation angles of the right shoulder in the YZ plane, the right shoulder in the XZ plane, and the right elbow, respectively. This indicates the pulse width of the servo motor corresponding to the rotation of the right shoulder in the YZ plane. This represents the pulse width of the servo motor corresponding to the rotation of the right shoulder in the XZ plane. This indicates the pulse width of the servo motor corresponding to the right elbow. The conversion formula between the finger bending angle data and the servo motor angle is: ， ， The above two formulas are the angle conversion formulas for the thumb and four fingers of the robotic hand, respectively. and To correspond to the maximum and minimum amplitude of the servo motor movement, This corresponds to the value of the finger bending angle data after being smoothed by the Kalman filter. The minimum value of β in the output. , The maximum value in the output β and The difference, or This corresponds to the difference between the maximum and minimum amplitude of the servo motor movement.

9. A vision-based human key point detection robotic hand synchronous control system, characterized in that, The vision-based human key point detection robotic hand synchronization control method according to any one of claims 1 to 8 includes: The image acquisition module is used to capture images of human arms and hands in real time via a camera; The key point detection module integrates the MediaPipe library and is used to obtain the key points of the human skeleton and the key points of the hand, as well as the corresponding data, based on the image. The coordinate processing module is used to extract the data of the key points of the human skeleton and the key points of the hand respectively, and then calculate the corresponding joint angle data and finger bending angle data. A Kalman filter module is used to smooth the output data of the coordinate processing module. The control signal conversion module is used to convert the smoothed joint angle data and the finger bending angle data into servo pulse widths and send the corresponding control signals via serial communication; and The robotic arm module is used to receive the control signals to achieve vision-based human key point detection and synchronous control of the robotic arm.

Images