Vision-based human body key point detection mechanical palm synchronous control method and system

Abstract

The invention provides a human body key point detection mechanical palm synchronous control method and system based on vision, and belongs to the technical field of robot control and human-computer interaction. The gesture recognition technology is used for conducting gesture estimation on the target arm, and key point data are captured and extracted. The key point data comprises joint position information of the arm and the hand, necessary input is provided for subsequent control, the data are processed, and the joint position information is converted into steering engine corner information capable of driving all joints of the mechanical arm. According to the invention, the Kalman filtering algorithm is adopted to filter the joint angle data, and then the filtered steering engine rotation angle information is transmitted to each joint point of the mechanical arm through the serial port communication technology, so that the precise control of the mechanical arm is realized, the stability and real-time performance of data transmission are ensured, and the reliability of the mechanical arm is improved. And the mechanical arm can quickly respond to the gesture change of a person.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot control and human-computer interaction, and in particular relates to a method and system for synchronous control of a robot palm based on vision-based key point detection of a human body. Background Art

[0002] With the continuous advancement of science and technology, robotics is increasingly being applied in numerous fields, including industry, healthcare, and services, becoming a key force driving the development of various industries. In this context, human-computer interaction technology, as a bridge between robots and humans, has become a key research area, as its development directly impacts the intelligence and effectiveness of robots.

[0003] Traditional robotic arm and hand control methods rely primarily on pre-set programs or manual remote control. Under pre-set program control, the robotic arm can only perform fixed tasks according to pre-set instructions, lacking adaptability and flexibility in complex and changing environments. For example, in industrial production, the diversity of products and adjustments to production processes can make it difficult for pre-set programs to meet demand. While manual remote control can achieve a certain degree of flexible control, it requires operator expertise, is complex, and can easily lead to fatigue after prolonged operation, impacting efficiency and accuracy.

[0004] The rise of human keypoint detection technology has brought new opportunities for controlling robotic arms and manipulators. This technology can accurately identify and locate key points on various parts of the human body, capturing information about the person's posture. Applying this technology to the control of robotic arms and manipulators enables more natural and intuitive human-machine interaction. The user simply makes the appropriate movements, and the system, by detecting changes in key points on the body, quickly interprets and converts these into control commands for the robotic arm and manipulator, allowing the robotic arm and manipulator to mimic human movements, significantly lowering the barrier to entry.

[0005] In the field of medical rehabilitation, robotic arms and robotic hands based on human body key point detection can assist medical staff in providing more accurate rehabilitation training for patients; in the field of education, they can be used to develop innovative teaching tools and improve teaching effectiveness; in the field of home services, they can help the elderly and disabled people operate smart devices more conveniently and improve their quality of life. Summary of the Invention

[0006] The present invention is based on the above background technology and aims to provide a method and system for synchronous control of a human body key point detection robot palm based on vision.

[0007] The present invention provides a method for synchronous control of a robot palm for detecting human key points based on vision, which has the following characteristics: S10, after installing the MediaPipe library, initializing the camera and capturing an image; S20, after using the BlazePose model to estimate the human posture in the image, detecting the human posture in the image, obtaining data of human skeleton key points, and marking and displaying the human skeleton key points thereon; S30, after using the palm_detection_SSD model to detect the ROI image and hand key points of the human hand in the image, using a regression network for inference to obtain data of the hand key points; S40, after respectively extracting the data of the human skeleton key points and the hand key points, calculating the corresponding joint angle data and finger bending angle data by using a vector method; S50, converting the joint angle data and the finger bending angle data into a servo pulse width or a servo angle, and sending a corresponding control signal through a communication serial port communication to drive the servo of the robot, thereby realizing synchronous control of the robot palm for detecting human key points based on vision.

[0008] In the vision-based human key point detection and robotic palm synchronization control method provided by the present invention, it can also have the following features: wherein, step S10 includes the following sub-steps: S11, using the pip command to install the MediaPipe library, importing the library through the import statement, and performing basic parameter configuration; S12, using OpenCV's cv2.VideoCapture(0) method to continuously capture the camera's image; S13, using the cv2.VideoCapture(0).isOpened method to verify and establish an image capture loop to ensure that the system can read image data in real time, wherein the default capture format of OpenCV is the original BGR format image frame; S14, using the cv2.cvtColor(image,cv2.COLOR_BGR2RGB) method to convert the original BGR format image frame into RGB format.

[0009] The vision-based human key point detection and robotic palm synchronization control method provided by the present invention may also have the following features: wherein, step S20 includes the following sub-steps: S21, using the BlazePose model to estimate the human posture of the preprocessed RGB format image; S22, using the holistic.process(image) method to detect the human posture estimated by the BlazePose model until the human posture is detected, marking the image as writable, and storing the data of the human skeleton key points; S23, converting the image from RGB format to BGR format, and displaying it through the cv.imshow method; S24, using the mp_drawing.draw_landmarks method to draw the human skeleton key points on the image.

[0010] The vision-based human key point detection and robotic palm synchronization control method provided by the present invention may also have the following features: wherein, in step S21, the BlazePose model includes a posture detector and a posture tracking network, the posture detector identifies the key points of the human skeleton, the posture tracking network predicts the coordinates of the key points of the human skeleton, confirms the presence of the human body in the current frame, and determines the ROI area of ​​the current frame; if the posture tracking network indicates that there is no human, the function of the posture detector is re-run in the next frame.

[0011] In the method for synchronous control of a robotic palm based on vision-based human key point detection provided by the present invention, the following features may also be provided: wherein, step S30 includes the following sub-steps: S31, adjusting the pre-processed RGB format image to a uniform resolution and converting it into a tensor format acceptable to TensorFlow; S32, after extracting features from the tensor format image using the palm_detection_SSD model, generating several anchor frames on feature maps of different scales of the image, and outputting corresponding classification score tensors and regression result tensors, wherein the classification score tensor indicates the probability of a palm existing in each anchor frame, and the regression result tensor contains the probability of each anchor frame existing in the palm frame. The bounding box offset of the frame and the coordinates of several palm key points are obtained; S33, the anchor frame with the highest classification score tensor is selected as the target palm candidate area, and after determining the palm detection frame according to its regression result tensor, the whole hand detection frame is calculated; S34, according to the four vertices of the hand detection frame, the hand ROI area is cropped out, and the area is rotated counterclockwise so that the palm center is located at the center of the image and the finger direction is parallel to the image edge, and the hand ROI image after centering and rotation correction is obtained; S35, the hand ROI image is scaled to a uniform pixel size; S36, the scaled ROI image is input into the regression network for inference, and the data of the hand key points is output.

[0012] The method for synchronous control of a robotic palm based on vision-based human key point detection provided by the present invention may also have the following features: wherein, in step S32, the scale of the anchor frame is set to: scale represents the scale of the anchor box, S min Indicates the minimum detection scale, S max Indicates the maximum detection scale, index = 0, 1, ..., num-1, num represents the hierarchical index of the current feature map. The palm_detection_SSD model extracts image features by using convolutional layers with filters of different sizes and biases, and then adds nonlinearity using activation layers. In step S33, the palm detection frame is obtained by the target palm candidate region and the decoding formula. The decoding formula is:

[0013] In the above formula, (x a ,y a ) represents the center coordinate of the target palm candidate area, w a and h a represents the width and height of the target palm candidate area, (x*, y*) represents the center coordinates of the palm detection frame, w* and h* represent the width and height of the palm detection frame, Represents the four offsets of the anchor frame represented by each anchor point in the regression result tensor; the hand detection frame is calculated by the following method: the palm center joint point and the first joint point of the middle finger in the palm key points are selected to calculate the direction vector V, and its rotation angle α with the vertical direction is calculated, the center of the palm detection frame is translated along the V direction to the palm center position, and then the width and height of the translated palm detection frame are multiplied by the empirical value to obtain the preliminary frame, and finally the four corner coordinates of the preliminary frame are rotated based on the angle α to form the hand detection frame.

[0014] In the vision-based human key point detection and robotic palm synchronization control method provided by the present invention, it can also have the following characteristics: wherein, step S40 includes the following sub-steps: S41, a three-dimensional coordinate system is established with the standing point of the person as the coordinate origin, the front of the person as the positive direction of the X-axis, the right direction as the positive direction of the Y-axis, and the top of the head as the positive direction of the Z-axis; S42, the key points of the human skeleton are extracted and the corresponding joint angle data are calculated; S43, after extracting the data of the hand key points, three arrays FINGER_MCP_INDEX, FINGER_PIP_INDEX and FINGER_DIP_INDEX are defined to store the key points of the proximal phalanges of the fingers, the key points of the distal phalanges of the fingers and the key points of the terminal phalanges of the fingers respectively; S44, according to the data of FINGER_MCP_INDEX, FINGER_PIP_INDEX and FINGER_DIP_INDEX, the five fingers are traversed to calculate the finger bending angle data of each finger.

[0015] The vision-based human key point detection and robotic palm synchronization control method provided by the present invention may also have the following features: in step S42, the holistic.process(image).pose_landmarks method is used to check whether the BlazePose model detects the human skeleton key points. If detected, the data of the human skeleton key points are stored in the RHL variable, and then the list pose_angle is initialized to store the calculated joint angle data; in step S43, the holistic.process(image).right_hand_landmarks method is used to check whether the hand key points are detected and obtained. If detected, the data of the hand key points are stored in the "RHL" variable, and then the list hand_angle is initialized to store the calculated finger bending angle data.

[0016] In the human body key point detection and synchronization control method for the manipulator provided by the present invention, the following features may also be provided: wherein, in step S50, the joint angle data and the finger bending angle data are smoothed using a Kalman filter, and the mapping method of the joint angle data and the servo pulse width is: PWM1 = 2000-(θ′1-90) ÷ 90×1500, PWM2 = 800-(θ′2-90) ÷ 90×1300, PWM3 = 900+θ′3 ÷ 180×1400, the above formula In the equation, θ'1, θ'2, and θ'3 are the corresponding values ​​of the joint angle data θ1, θ2, and θ3 after smoothing by Kalman filtering. The joint angle data θ1, θ2, and θ3 are the rotation angles of the right shoulder in the YZ plane, the right shoulder in the XZ plane, and the right elbow, respectively. PWM1 represents the pulse width of the servo corresponding to the rotation of the right shoulder in the YZ plane, PWM2 represents the pulse width of the servo corresponding to the rotation of the right shoulder in the XZ plane, and PWM3 represents the pulse width of the servo corresponding to the right elbow. The conversion formula between the finger bending angle data and the servo angle is:

[0017] M1-(β-m1)÷n1×l1,

[0018] M2+(β-m2)÷n2×l2,

[0019] The above two formulas are the angle conversion formulas of the thumb and four fingers of the manipulator, respectively. Among them, M1 and M2 are the maximum and minimum amplitudes of the corresponding servo movement, β is the corresponding value of the finger bending angle data after smoothing by Kalman filtering, m is the minimum value of the output β, n1 and n2 are the differences between the maximum value of the output β and m, and l1 or l2 is the difference between the maximum and minimum amplitudes of the corresponding servo movement.

[0020] The present invention also provides a vision-based human key point detection robotic palm synchronization control system, which has the following characteristics: it uses any of the aforementioned vision-based human key point detection robotic palm synchronization control methods, including: an image acquisition module, used to capture images containing human arms and hands in real time through a camera; a key point detection module, integrated with a MediaPipe library, used to obtain human skeleton key points and hand key points and corresponding data of the human body based on the image; a coordinate processing module, used to extract the data of human skeleton key points and hand key points respectively, and calculate the corresponding joint angle data and finger bending angle data; a Kalman filter module, used to smooth the output data of the coordinate processing module; a control signal conversion module, used to convert the smoothed joint angle data and finger bending angle data into a servo pulse width and send the corresponding control signal through a communication serial port; and a robotic arm module, used to receive the control signal to realize vision-based human key point detection robotic palm synchronization control.

[0021] Functions and effects of the invention

[0022] Based on the needs of human-computer interaction, this invention comprehensively utilizes advanced image processing technology, machine learning algorithms, filtering technology and communication technology to achieve precise control of the robotic arm.

[0023] In implementing the control system, the present invention first uses gesture recognition technology to estimate the target arm's posture, capturing and extracting key point data. This key point data includes information about the joint positions of the arm and hand, providing the necessary input for subsequent control. The system then processes this data through a series of algorithms, converting the joint position information into servo angle information that can drive each joint of the robotic arm.

[0024] To ensure stable and real-time output, the present invention employs a Kalman filter algorithm to filter joint angle data. This filter effectively estimates the system's true state from noisy measurements, reducing random fluctuations caused by sensor noise and other interfering factors, resulting in a smoother and more reliable system output.

[0025] This invention uses serial communication technology to transmit filtered servo angle information to each joint of the robotic arm, achieving precise control of the robotic arm. Serial communication technology ensures stable and real-time data transmission, allowing the robotic arm to quickly respond to changes in human gestures.

[0026] The entire process of this invention involves a complex series of operations, from data acquisition to processing, and from decision-making to execution. The design of this invention fully considers the requirements of real-time performance, accuracy, and stability. Through actual arm-following and hand-following experiments, it has demonstrated excellent performance. The robotic arm can capture and recognize human postures in real time, accurately simulating continuous arm postures and specific hand gestures, and the system responds quickly and has excellent stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flowchart of a method for synchronously controlling a human body key point detection robot palm based on vision in an embodiment of the present invention;

[0028] Figure 2 It is a flowchart of image capture and human skeleton key point detection in steps S10 to S20 of an embodiment of the present invention;

[0029] Figure 3 It is the output display image after marking and drawing the key points of the human skeleton in step S20 of the embodiment of the present invention;

[0030] Figure 4 Schematic diagram of input and output of the palm_detection_SSD model in step S30 of an embodiment of the present invention;

[0031] Figure 5 is an output display diagram of the palm_detection_SSD model in step S33 of an embodiment of the present invention;

[0032] Figure 6 1 is an output display diagram of the palm_detection_SSD model after the regression network diagnosis in step S36 of the embodiment of the present invention;

[0033] Figure 7 It is a flowchart of the key point data processing of the right arm in step S42 of an embodiment of the present invention;

[0034] Figure 8 It is the marking of the key points of the human skeleton and the key points of the palm in the embodiment of the present invention;

[0035] Figure 9 is a flowchart of the right-hand key point data processing in step S44 of an embodiment of the present invention;

[0036] Figure 10 is a schematic diagram of the bending angle data of a joint after Kalman filtering in step S51 of an embodiment of the present invention;

[0037] Figure 11Schematic diagram of the connection relationship of a vision-based human key point detection robot palm synchronization control system in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate the vision-based human key point detection robot palm synchronization control method and system of the present invention.

[0039] <Example>

[0040] Figure 1 This is a flowchart of a method for synchronously controlling a robotic palm based on vision-based detection of key points of a human body in an embodiment of the present invention.

[0041] like Figure 1 As shown, this embodiment provides a method for synchronous control of a robotic palm based on human key point detection, comprising the following steps:

[0042] S10, after installing the MediaPipe library, initializes the camera and captures images, including the following sub-steps S11 to S14:

[0043] In S11, use the pip command to install the MediaPipe library, import the library using the import statement, and configure basic parameters.

[0044] The MediaPipe library is a powerful toolset that integrates a variety of machine learning vision algorithms specifically designed for gesture recognition and pose estimation tasks.

[0045] S12, use OpenCV's cv2.VideoCapture(0) method to continuously capture the camera image.

[0046] S13, using the cv2.VideoCapture(0).isOpened method to verify and establish an image capture loop to ensure that the system can read image data in real time. The default capture format of OpenCV is the original BGR format image frame.

[0047] S14, convert the original BGR format image frame to RGB format using the cv2.cvtColor(image,cv2.COLOR_BGR2RGB) method.

[0048] S20, human skeleton key point detection: After using the BlazePose model to estimate the human posture in the image, detect the human posture in the image, obtain the data of the human skeleton key points, and mark and draw the human skeleton key points on it, including the following sub-steps S21 to S24:

[0049] S21, use the BlazePose model to estimate the human posture of the preprocessed RGB format image.

[0050] The BlazePose model is a lightweight convolutional neural network architecture optimized for real-time inference on mobile devices for human pose estimation. It consists of a lightweight pose detector and a pose tracking network.

[0051] The pose detector identifies key points on the human skeleton, while the pose tracking network predicts the coordinates of these key points, confirms the presence of a person in the current frame, and determines the ROI (Region of Interest) for the current frame. If the pose tracking network indicates that no person is present, the system reruns the pose detector on the next frame to continue pose estimation.

[0052] S22, in the holistic module of MediaPipe, use the holistic.process(image) method to detect the human posture estimated by the BlazePose model until the human posture is detected, mark the image as writable, and store the data of the human skeleton key points.

[0053] S23, after converting the image from RGB format back to BGR format, it is displayed in the OpenCV environment through the cv.imshow method, so as to intuitively show the results of the pose estimation and provide a visual reference for subsequent further analysis and processing. Figure 3 shown.

[0054] S24, use the mp_drawing.draw_landmarks method to draw the human skeleton key points on the image.

[0055] S30, hand detection and hand key point detection: After using the palm_detection_SSD model to detect the ROI image and hand key points of the human hand in the image, a regression network is used for inference to obtain the data of the hand key points, which specifically includes the following sub-steps S31 to S36:

[0056] S31, resize the RGB format image to a uniform resolution of 128×128 and convert it to a tensor format acceptable to TensorFlow.

[0057] Specifically, it is a four-bit tensor with a size of 1×128×128×3, where 1 represents the batch size, 128×128 is the resolution of the image, and 3 corresponds to the RGB color channels, as shown in Figure 4 As shown in part (a) of .

[0058] S32, processing the image in tensor format to obtain a classification score tensor and a regression result tensor, including the following sub-steps S32-1 to S32-3:

[0059] S32-1, use the palm_detection_SSD model to extract features from images in tensor format.

[0060] In the convolutional layers of the palm_detection_SSD model, the input tensor is processed through multiple convolutional layers, which use filters of varying sizes and biases to extract image features. Convolution operations are typically followed by an activation layer, which adds nonlinearity to the output of the convolutional layer, enhancing the model's expressiveness.

[0061] S32-2, such as Figure 4 As shown in part (b) of the figure, the palm_detection_SSD model generates 896 anchor boxes based on feature maps of different scales in the image to predict the location of the palm. The size and number of channels of these feature maps are detailed in the figure. For example, 1×16×16×128 represents a 16×16 grid with 128 feature channels per grid point.

[0062] Among them, the scale of the anchor box is set to:

[0063]

[0064] In the above formula, scale represents the scale of the anchor box, S min Indicates the minimum detection scale, S max Indicates the maximum detection scale, index = 0, 1, ..., num-1, num represents the level index of the current feature map.

[0065] According to the above formula, these anchor boxes are defined on feature maps of two different scales: one part of the anchor boxes is based on the feature map of 16×16 resolution, using two different scales (0.12 and 0.23 respectively); the other part of the anchor boxes is generated on the feature map of 8×8 resolution, using six different scales (0.35, 0.45, 0.61, 0.74, 0.87 and 1 respectively).

[0066] S32-3, output the corresponding classification score tensor and regression result tensor:

[0067] After the palm_detection_SSD model completes inference, it outputs two tensors:

[0068] A 1×896×1 classification score tensor containing the classification scores of the 896 anchor points. These scores range from 0 to 1, reflecting the probability of a palm being present in each anchor point (the closer the value is to 1, the greater the probability of detecting a palm). In the experiment, the classification score threshold is set to 0.60. If the score is lower than this threshold, it is considered that no palm is detected in the corresponding anchor point.

[0069] The other is a 1×896×18 regression result tensor, which contains the regression results of each anchor point, where each anchor point has 4 values ​​​​to represent the offset of the bounding box And the coordinates of 7 palm key points (each key point has two coordinates, x and y), a total of 18 values. These offsets and key point coordinates help to accurately locate the position of the palm.

[0070] After the palm_detection_SSD model completes inference and generates a classification result tensor and a regression result tensor, the next step is to decode these results to determine the precise location of the palm:

[0071] S33, select the anchor frame with the highest classification score tensor as the target palm candidate region, determine the palm detection frame based on its regression result tensor, and then calculate the whole hand detection frame, including the following sub-steps S33-1 to S33-2:

[0072] S33-1, determine the palm detection frame palm_rect:

[0073] Since the premise of this embodiment is to detect only one hand in the image, the decoding process will select the anchor frame with the highest classification score from the 896 anchor frames as the candidate region of the target palm. The center coordinate of the candidate region is marked as (x a ,y a ), width and height are denoted as w a and h a , and the center coordinates of the palm detection frame of the actual final detection result are marked as (x*, y*), and the width and height are recorded as w* and h* respectively. The calculation formula (decoding formula) is as follows:

[0074]

[0075] After the above decoding formula, and based on the classification and regression results provided by the palm_detection_SSD model, the palm detection frame palm_rect is obtained. Its center coordinates are given by (x*, y*), and its width and height are w* and h* respectively. Among them, the palm detection frame palm_rect in this step is as follows Figure 5As shown in the blue box in , there are seven white key points displayed therein, which are the 7 palm key points output by the palm_detection_SSD model in step S32-3.

[0076] S33-2, calculate the entire hand detection frame hand_rect:

[0077] First, select the center joint point of the palm and the first joint point of the middle finger from the palm detection frame palm_rect, and use the coordinates of these two points to calculate the vector V( Figure 5 The angle between this vector and the y-axis (vertical direction) is calculated to determine the rotation angle α of the palm.

[0078] Then, the center coordinates of the palm detection frame palm_rect are translated along the direction of vector V to the approximate position of the palm center.

[0079] Next, multiply the width and height of the rectangle by an empirical value multiple (usually 2.8) to obtain Figure 5 Then, according to the calculated rotation angle α, the coordinates of the four corners of the orange box are transformed to form Figure 5 The red box in the figure represents the entire hand detection box hand_rect, which includes not only the palm but also all fingers, ensuring that all important hand features can be covered in the subsequent key point detection.

[0080] S34: Based on the four vertices of the hand detection frame hand_rect, the hand ROI area is cropped and the area is rotated counterclockwise by an angle of α so that the palm is located at the center of the image and the direction of the fingers is parallel to the edge of the image, thereby obtaining a centered and rotation-corrected hand ROI image.

[0081] S35, scaling the hand ROI image (i.e., the red box of the hand) to a uniform size of 224×224 pixels.

[0082] S36: Input the scaled ROI image into the regression network for inference, and output the data of the hand key points. A total of three results are output:

[0083] (1) A 1×1 classification result, which gives a score indicating the confidence level of the detected key points, that is, the reliability of the key points.

[0084] (2) 1×1 classification results, used to distinguish whether the detected hand is the left hand or the right hand. When the output value is close to 1, it means that the detected hand is more likely to be the right hand; if the value is close to 0, it means that it is more likely to be the left hand.

[0085] (3) A 1×63 regression result, which contains the three-dimensional coordinate information of 21 skeleton points in the current detection image.

[0086] like Figure 6 As shown in , these coordinates include values ​​in three directions: X, Y, and Z. The X and Y coordinates are normalized and range between 0 and 1. They represent the position of the skeleton point in the original hand image, while the Z coordinate provides the distance of the skeleton point relative to the hand key point 0 at the bottom of the palm near the wrist (as shown in Figure 6 Part a and Figure 8 The Z coordinate of keypoint 0 is always set to 0.0 as the reference point, while the Z coordinates of the remaining keypoints indicate whether they are closer (positive values) or farther (negative values) from the camera relative to point 0. This gives us the 3D data for all the keypoints of the hand.

[0087] S40, after extracting the data of the key points of the human skeleton and the key points of the hand, calculates the corresponding joint angle data and finger bending angle data by vector method, including the following sub-steps S41 to S44:

[0088] S41, establish a three-dimensional coordinate system with the person's standing point as the coordinate origin, the front of the person as the positive direction of the X axis, the right direction as the positive direction of the Y axis, and the top of the head as the positive direction of the Z axis.

[0089] S42, such as Figure 7 As shown, extracting the key points of the human skeleton and calculating the corresponding joint angle data includes the following sub-steps S42-1 to S42-3:

[0090] S42-1, check whether human skeleton key points are detected through the holistic.process(image).pose_landmarks method, and if detected, store these human skeleton key points in the RHL variable.

[0091] S42-2, calculate joint angle data:

[0092] (1) Calculate the joint angle data of the right shoulder in the YZ plane.

[0093] Three key points are extracted from RHL, namely Figure 8 In part a, the skeletal key points 11, 12, and 14 correspond to the spatial coordinates of the left shoulder, right shoulder, and right elbow. Define a function called "calculate_pose_angle2dyz(landmark1, landmark2, landmark3)" to calculate the angle between the three points.

[0094] Calculate the joint angle of the right shoulder in the YZ plane. Given the spatial coordinates of three points, we can get two vectors using the vector formula:

[0095]

[0096] in, is the vector from point i to point j, (x i ,y i ,z i ) and (x j ,y j ,z j ) are the three-dimensional coordinates of the two points. Let the three key points 11, 12, and 14 be A(x1,y1,z1), B(x2,y2,z2), and C(x3,y3,z3), respectively, and we can get two vectors:

[0097]

[0098] Since we are projecting onto the YZ plane, the X coordinate is 0. Therefore, the vector:

[0099]

[0100] After obtaining the vectors, use the vector dot product formula to calculate the angle θ between the two vectors:

[0101]

[0102] in, and For two vectors, the dot product for:

[0103]

[0104] vector and The modules are:

[0105]

[0106]

[0107] Substituting into equation 1, equation 2, equation 3, and equation 4, we can obtain and The cosine value of the angle θ1 is:

[0108]

[0109] but and The value of the angle θ1

[0110]

[0111] (2) Calculate the joint angle data of the right shoulder in the XZ plane.

[0112] That is, if you want to project it onto the XZ plane, the Y coordinate is 0. The vector formula of , we get the vector:

[0113]

[0114] Substituting into equation 1, equation 2, equation 3, and equation 4, we get and The absolute value of the cosine of the angle θ2 is

[0115]

[0116] but and The value of the angle θ2

[0117]

[0118] Since the human arm's range of motion in the XZ plane is 0° to 180°, we limit the servo that controls the right shoulder's XZ motion accordingly. If the range exceeds 180°, the calculation is performed as if it were 180°. The remaining calculation process is similar to that for the right shoulder's YZ plane motion, ultimately resulting in the joint angle data.

[0119] (3) Calculate the joint angle data of the right elbow movement.

[0120] Three key points are extracted from RHL, namely Figure 8 The key points 12, 14 and 16 of the human skeleton in part a correspond to the spatial coordinates of the right shoulder, right elbow and right wrist. The calculation method is similar to the above. Let the three key points 12, 14 and 16 be B(x2,y2,z2), C(x3,y3,z3) and D(x4,y4,z4) respectively. The vector formula can be used to obtain two vectors:

[0121]

[0122] Substituting into equation 1, equation 2, equation 3, and equation 4, we get and The cosine of the angle θ3 is

[0123]

[0124] but and The value of the angle θ3

[0125] θ3=arccos(cos(θ3))

[0126] At this point, the joint angle data of the entire arm is obtained.

[0127] S42-3, initialize the list pose_angle to store the joint angle data for subsequent calculation.

[0128] S43, after extracting the data of the hand key points, defines three arrays: FINGER_MCP_INDEX, FINGER_PIP_INDEX, and FINGER_DIP_INDEX, which respectively store the key points of the proximal phalanges of the fingers, the key points of the distal phalanges of the fingers, and the key points of the distal phalanges of the fingers. This includes the following sub-steps S43-1 to S43-2:

[0129] S43-1, checks whether the right hand landmarks are detected in the hand landmarks using the holistic.process(image).right_hand_landmarks method. If detected, these landmarks are stored in the RHL variable. Next, the hand_angle list is initialized to store the calculated finger bend angle data.

[0130] S43-2, define three arrays FINGER_MCP_INDEX, FINGER_PIP_INDEX, FINGER_DIP_INDEX, respectively store Figure 8 The following key points in part b: (1) key points of the proximal phalangeal joint (MCP) of the fingers, namely key points 1, 5, 9, 13 and 17; (2) key points of the distal phalangeal joint (PIP) of the fingers, namely key points 2, 6, 10, 14 and 18; (3) key points of the distal phalangeal joint (DIP), namely key points 3, 7, 11, 15 and 19.

[0131] S44 , according to the data of FINGER_MCP_INDEX, FINGER_PIP_INDEX, and FINGER_DIP_INDEX, traverse the five fingers to calculate the finger bending angle data of each finger.

[0132] When calculating the degree of finger bending, the coordinate changes of the three key joints MCP, PIP and DIP are directly related to the bending action of the finger, providing key information for accurate gesture capture and manipulator control. Therefore, the degree of bending of each finger can be calculated based on the angle formed by them. The calculation method is as follows Figure 9 As shown:

[0133] Calculate the degree of thumb bending and set a loop to traverse the five fingers. Extract the three proximal, distal and terminal phalangeal key points of the thumb from RHL, which are Figure 8 Define a function "calculate_finger_angle(MCP,PIP,DIP)" to calculate the angle between the three points in the space.

[0134] set up Figure 8 The three key points 1, 2, and 3 in part b are M1 (x5, y5, z5), P1 (x6, y6, z6), and D1 (x7, y7, z7). The vector formula can be used to obtain two vectors:

[0135]

[0136]

[0137] Substituting into equation 1, equation 2, equation 3, and equation 4, we get and The cosine value of the angle θ4 is:

[0138]

[0139] Then the value of the angle θ4, i.e. the bending degree of the thumb, can be obtained. Similarly, the bending angle data of the remaining four fingers can be obtained by traversing the loop.

[0140] S50, after smoothing the joint angle data and the finger bending angle data using a Kalman filter, converts the corresponding data into a servo pulse width and sends a corresponding control signal through a serial communication port to drive the servo of the manipulator, thereby realizing vision-based human body key point detection and synchronous control of the manipulator palm, including the following sub-steps:

[0141] S51, use Kalman filtering to smooth the joint angle data and finger bending angle data:

[0142] The palm_detection_SSD model may be affected by various noises when capturing hand keypoints, such as camera noise, hand jitter, and ambient light fluctuations. This noise can cause the acquired data to be less smooth and may even contain outliers, directly affecting the accuracy of the conversion from gestures to servo control signals. This step uses the Kalman filter algorithm to optimize response accuracy.

[0143] The Kalman filter is a state estimation algorithm for linear dynamic systems. It effectively estimates the system's true state from noisy measurements. This goal is achieved through two main steps: prediction, which predicts the state at the next moment based on the system's dynamic model. The palm_detection_SSD model predicts the positions of gesture keypoints in the next frame. The second step is update. When new measurement data arrives, the Kalman filter compares the predicted state with the actual observed data and adjusts the prediction based on the difference to obtain a more accurate estimate.

[0144] This step creates two Kalman filter instances, specifying a state vector dimension of 4 and an observation vector dimension of 4. These are used to filter the servo angle data of the manipulator and manipulator, respectively, to reduce noise and improve control accuracy.

[0145] Since the human arm movement is assumed to be slow, the hand moves approximately in a uniform linear motion in front of the camera, which can be regarded as a linear system. The prediction phase begins by estimating the current state It is calculated based on the state estimation and process model of the previous moment. The state vector usually contains all the internal states of the system, such as the gesture angles of all fingers. The process model can be expressed as the state transfer matrix F k , which describes how the state evolves over time. Therefore, the predicted state can be expressed as:

[0146]

[0147] Among them, k represents the current moment.

[0148] This step assumes that the angle information does not change during the prediction process, that is, the rate of change of the angle information during state transfer is 0. Based on this, the state transfer matrix kf.F is set as follows:

[0149] kf.F=np.array([[1,0,0,0,0]

[0150] [0,1,0,0,0]

[0151] [0,0,1,0,0]

[0152] [0,0,0,1,0]

[0153] [0,0,0,0,1]])

[0154] As shown above, the Kalman filter state transfer matrix kf.F describes the relationship between the current state and the state at the next moment.

[0155] Then set the measurement matrix kf.H, which describes the relationship between the state variables and the observed variables. Here, it is assumed that the angle information can be directly observed, that is, the angle information in the measurement matrix is ​​directly mapped to the state vector:

[0156] kf.H=np.array([[1,0,0,0,0]

[0157] [0,1,0,0,0]

[0158] [0,0,1,0,0]

[0159] [0,0,0,1,0]

[0160] [0,0,0,0,1]])

[0161] At the same time, the Kalman filter also predicts the error covariance matrix P k|k-1 , which describes the uncertainty of the predicted state. The error covariance matrix P k|k-1 The update formula is:

[0162]

[0163] Among them, Q k is the process noise covariance matrix, which represents the uncertainty of the system in time evolution, Indicates F k The transposed matrix of .

[0164] The Kalman filter uses the new measurement data z k To correct the prediction. Measurement model H k Describes how the state vector is mapped to the measurement vector. The measurement vector may be directly obtained from the sensor, such as the observation coordinates of the key points of the hand.

[0165] The measurement update consists of two main steps: calculating the Kalman gain K k and update the state estimate.

[0166] The Kalman gain is the measurement noise covariance matrix R k and the prediction error covariance matrix P k|k-1 The function is calculated as

[0167]

[0168] The Kalman gain determines the degree to which the measurement data affects the state estimate. Then, the Kalman gain is used to update the state estimate.

[0169]

[0170] in, is the corrected state estimate, which combines the predicted information and the new measurement data. Finally, the error covariance matrix is ​​updated:

[0171] P k|k =(IK k H k )P k|k-1

[0172] In the above formula, I represents the identity matrix.

[0173] The update of the error covariance matrix reflects how the new measurement data reduces the uncertainty of the prediction.

[0174] In this step, it is assumed that the process noise is constant and independent of each other. In the update phase, the process noise covariance matrix Q is set k and the measurement noise covariance matrix R k The parameters are 0.01 and 0.1 respectively. The initial state vector and initial covariance matrix are set to np.array([0,0,0,0,0]) and 10 respectively. The settings of the second Kalman filter are the same as above.

[0175] During the filtering process, the "kf.predict" method is used to predict the next state, the "kf.update" method is used to update the current state, the observed value is used for correction, and finally the "kf.x" method is used to output the filtered value. Throughout this process, the Kalman filter continuously iterates between prediction and update to provide a continuous estimate of the system state. As long as the system continues to repeat these two steps, the system state will continue to track the observed value and have a certain filtering effect. This recursive nature makes the Kalman filter well-suited for gesture recognition and robotic control systems, where the data flow is continuous and real-time.

[0176] Figure 10 Schematic diagram of the bending angle data of a joint after Kalman filtering in step S51 of the embodiment of the present invention. Figure 10 As shown, it can be seen that the jitter phenomenon of the joint angle data obtained after the above steps has been significantly improved.

[0177] The Kalman filter uses its mathematical model to reduce random fluctuations caused by sensor noise and other interfering factors. The filter also produces smoother output data, helping to avoid erroneous operations caused by sudden data changes. Furthermore, the Kalman filter provides a stability analysis, helping the system maintain stability in the face of uncertainty. It is suitable for real-time systems, has a fixed computational complexity, and can execute quickly, which is crucial for real-time gesture recognition and robotic arm control. By using the Kalman filter, gesture recognition systems can provide more accurate and reliable hand posture information.

[0178] S52, converts the corresponding data into the servo pulse width or servo control signal:

[0179] (1) Use analytical methods to convert the joint angle data obtained in step S42-2 into the corresponding pulse width of the servo rotation angle. Design a corresponding formula based on the corresponding relationship between the PWM signal and the servo angle:

[0180] PWM1=2000-(θ′1-90)÷90×1500

[0181] PWM2=800-(θ′2-90)÷90×1300

[0182] PWM3=900+θ′3÷180×1400

[0183] In the above formula, θ'1, θ'2, and θ'3 are the θ1, θ2, and θ3 data after being smoothed by the Kalman filter in step S51. PWM1 represents the pulse width of the servo corresponding to the rotation of the right shoulder in the YZ plane, PWM2 represents the pulse width of the servo corresponding to the rotation of the right shoulder in the XZ plane, and PWM3 represents the pulse width of the servo corresponding to the right elbow.

[0184] (2) The finger bending angle data obtained in step S44 is normalized to convert it into a control signal for the servo. The conversion formula between the angle and the servo angle is obtained based on the experiment:

[0185] M1-(β-m1)÷n1×l1

[0186] M2+(β-m2)÷n2×l2

[0187] The above two equations are the angle conversion formulas for the thumb and four fingers of the manipulator, respectively. Here, M1 and M2 are the maximum and minimum amplitudes of the corresponding servo motion, β is the corresponding value of the finger bending angle data θ4 after smoothing by the Kalman filter in step S51, m is the minimum value in the output β, n1 and n2 are the differences between the maximum value in the output β and m, and l1 or l2 is the difference between the maximum and minimum amplitudes of the corresponding servo motion.

[0188] S53, sending corresponding control signals through the serial communication port to drive the servo of the manipulator to achieve vision-based human body key point detection and synchronous control of the manipulator palm.

[0189] Figure 11 Schematic diagram of the connection relationship of a vision-based human key point detection robot palm synchronization control system in an embodiment of the present invention.

[0190] like Figure 11As shown, this embodiment also provides a vision-based human key point detection robotic palm synchronization control system, which uses a vision-based human key point detection robotic palm synchronization control method provided by this embodiment, including an image acquisition module 10, a key point detection module 20, a coordinate processing module 30, a Kalman filter module 40, a control signal conversion module 50 and a robotic arm module 60.

[0191] The image acquisition module 10 is used to capture images containing human arms and hands in real time through a camera.

[0192] The key point detection module 20 is integrated with the MediaPipe library, and is used to obtain the human skeleton key points and hand key points and corresponding data of the human body according to the image captured by the image acquisition module 10 according to the method of steps S10 to S30.

[0193] The coordinate processing module 30 is used to extract the data of the key points of the human skeleton and the key points of the hand according to the method of step S40, and then calculate the corresponding joint angle data and finger bending angle data.

[0194] The Kalman filter module 40 is used to smooth the output data of the coordinate processing module 30 .

[0195] The control signal conversion module 50 is used to convert the smoothed joint angle data and finger bending angle data into a servo pulse width according to the method of step S50 and send a corresponding control signal through the communication serial port.

[0196] The manipulator module 60 is used to receive the control signal sent by the control signal conversion module 50 to achieve vision-based human body key point detection and manipulator palm synchronization control.

[0197] Those skilled in the art will appreciate that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A vision-based human key point detection and robotic palm synchronization control method, characterized in that: The following steps are involved: S10, after installing the MediaPipe library, initialize the camera and capture images; S20, after estimating the human posture in the image using the BlazePose model, detecting the human posture in the image, obtaining data of key points of the human skeleton, and marking and displaying the key points of the human skeleton on the image; S30, after detecting the ROI image and the key points of the hand of the human body in the image using the palm_detection_SSD model, inferring using a regression network to obtain data of the key points of the hand; S40, after extracting the data of the human skeleton key points and the data of the hand key points respectively, calculating the corresponding joint angle data and finger bending angle data by using a vector method; S50, converting the joint angle data and the finger bending angle data into a servo pulse width or a servo angle and sending a corresponding control signal through a serial communication port to drive the servo of the manipulator, thereby realizing vision-based human body key point detection and synchronous control of the manipulator palm.

2. The method for synchronous control of a human body key point detection robot palm based on vision according to claim 1, characterized in that: in, Step S10 includes the following sub-steps: S11. Use the pip command to install the MediaPipe library, import the library using the import statement, and configure basic parameters. S12, using OpenCV's cv2.VideoCapture(0) method to continuously capture the camera image; S13, using the cv2.VideoCapture(0).isOpened method to verify and establish an image capture loop to ensure that the system can read image data in real time. The default capture format of OpenCV is the original BGR format image frame; S14, convert the original BGR format image frame to RGB format using the cv2.cvtColor(image,cv2.COLOR_BGR2RGB) method.

3. The method for synchronous control of a robot palm based on vision-based human key point detection according to claim 1, characterized in that: in, Step S20 includes the following sub-steps: S21, using the BlazePose model to estimate the human body posture of the pre-processed RGB image; S22, using the holistic.process(image) method to detect the human posture estimated by the BlazePose model until the human posture is detected, marking the image as writable, and storing the data of the key points of the human skeleton; S23, converting the image from RGB format to BGR format, and displaying it using the cv.imshow method; S24, using the mp_drawing.draw_landmarks method to draw key points of the human skeleton on the image.

4. The method for synchronous control of a robot palm based on vision-based human key point detection according to claim 3, characterized in that: in, In step S21, the BlazePose model includes a pose detector and a pose tracking network. The posture detector identifies key points of the human skeleton, The posture tracking network predicts the coordinates of the key points of the human skeleton, confirms the existence of the human body in the current frame, and determines the ROI area of ​​the current frame. If the posture tracking network indicates that there is no human, the function of the posture detector is re-run in the next frame.

5. The method for synchronous control of a human body key point detection robot palm based on vision according to claim 1, Its characteristics are: in, Step S30 includes the following sub-steps: S31, adjusting the pre-processed RGB format image to a uniform resolution and converting it into a tensor format acceptable to TensorFlow; S32, after using the palm_detection_SSD model to extract features from the image in tensor format, generate several anchor boxes on the feature maps of the image at different scales, and output corresponding classification score tensors and regression result tensors. The classification score tensor indicates the probability of a palm in each anchor box, and the regression result tensor contains the bounding box offset of each anchor box and the coordinates of several palm key points. S33, selecting the anchor frame with the highest classification score tensor as the target palm candidate region, determining the palm detection frame based on its regression result tensor, and then calculating the whole hand detection frame; S34, cropping a hand ROI region based on the four vertices of the hand detection frame, and performing a counterclockwise rotation on the region so that the palm is located at the center of the image and the finger directions are parallel to the image edges, thereby obtaining a centered and rotation-corrected hand ROI image; S35, scaling the hand ROI image to a uniform pixel size; S36: Input the scaled ROI image into the regression network for inference, and output the data of the hand key points.

6. The method for synchronous control of a robot palm based on vision-based human key point detection according to claim 5, characterized in that: in, In step S32, the scale of the anchor box is set to: scale represents the scale of the anchor box, S min Indicates the minimum detection scale, S max Indicates the maximum detection scale, index=0,1,...,num-1, num represents the level index of the current feature map, The palm_detection_SSD model extracts features from the image using convolutional layers with filters of different sizes and biases, and then adds nonlinearity using activation layers. In step S33, the palm detection frame is obtained by the target palm candidate area and a decoding formula, and the decoding formula is: In the above formula, (x a ,y a ) represents the center coordinate of the target palm candidate area, w a and h a represents the width and height of the target palm candidate area, (x*, y*) represents the center coordinates of the palm detection frame, w* and h* represent the width and height of the palm detection frame, Represents the four offsets of the anchor box represented by each anchor point in the regression result tensor; The hand detection frame is calculated in the following manner: the palm center joint point and the first joint point of the middle finger are selected from the palm key points to calculate the direction vector V, and its rotation angle α with the vertical direction is calculated, the center of the palm detection frame is translated along the V direction to the palm center position, and then the width and height of the translated palm detection frame are multiplied by empirical values ​​to obtain a preliminary frame, and finally the four corner coordinates of the preliminary frame are rotated based on the angle α to form the hand detection frame.

7. The method for synchronous control of a human body key point detection robot palm based on vision according to claim 1, Its characteristics are: in, Step S40 includes the following sub-steps: S41, establish a three-dimensional coordinate system with the person's standing point as the coordinate origin, the person's front as the positive direction of the X axis, the right side as the positive direction of the Y axis, and the top of the head as the positive direction of the Z axis; S42, extracting the key points of the human skeleton and calculating the corresponding joint angle data; S43, after extracting the data of the hand key points, define three arrays FINGER_MCP_INDEX, FINGER_PIP_INDEX and FINGER_DIP_INDEX to store the key points of the proximal phalanges of the fingers, the key points of the distal phalanges of the fingers and the key points of the distal phalanges of the fingers respectively; S44 , according to the data of FINGER_MCP_INDEX, FINGER_PIP_INDEX, and FINGER_DIP_INDEX, traverse the five fingers to calculate the finger bending angle data of each finger.

8. The method for synchronous control of a robot palm based on vision-based human key point detection according to claim 7, characterized in that: in, In step S42, the holistic.process(image).pose_landmarks method is used to check whether the BlazePose model detects the human skeleton key points. If detected, the data of the human skeleton key points are stored in the RHL variable, and then the list pose_angle is initialized to store the calculated joint angle data; In step S43, the holistic.process(image).right_hand_landmarks method is used to check whether the hand key points are detected and obtained. If detected, the data of the hand key points are stored in the "RHL" variable, and then the list hand_angle is initialized to store the calculated finger bending angle data.

9. The method for synchronous control of a robot palm based on vision-based human key point detection according to claim 7, characterized in that: in, In step S50, Kalman filtering is also used to smooth the joint angle data and the finger bending angle data. The mapping method between the joint angle data and the servo pulse width is: PWM1=2000-(θ′1-90)÷90×1500 PWM2=800-(θ′2-90)÷90×1300 PWM3=900+θ′3÷180×1400 In the above formula, θ'1, θ'2, and θ'3 are the corresponding values ​​of the joint angle data θ1, θ2, and θ3 after being smoothed by the Kalman filter. The joint angle data θ1, θ2, and θ3 are the rotation angle of the right shoulder in the YZ plane, the rotation angle of the right shoulder in the XZ plane, and the rotation angle of the right elbow, respectively. PWM1 represents the pulse width of the servo corresponding to the rotation of the right shoulder in the YZ plane, PWM2 represents the pulse width of the servo corresponding to the rotation of the right shoulder in the XZ plane, and PWM3 represents the pulse width of the servo corresponding to the right elbow. The conversion formula between the finger bending angle data and the steering gear angle is: M1-(β-m1)÷n1×l1 M2+(β-m2)÷m2×l2 The above two formulas are the angle conversion formulas of the thumb and four fingers of the manipulator, respectively, where M1 and M2 are the maximum and minimum amplitudes of the corresponding servo movement, β is the corresponding value of the finger bending angle data after smoothing by the Kalman filter, m is the minimum value in the output β, n1 and n2 are the differences between the maximum value in the output β and m, and l1 or l2 is the difference between the maximum and minimum amplitudes of the corresponding servo movement.

10. A vision-based human key point detection and robotic palm synchronization control system, characterized in that: The method for synchronous control of a robot palm using vision-based human key point detection according to any one of claims 1 to 9 comprises: An image acquisition module, used for capturing images containing human arms and hands in real time through a camera; A key point detection module, integrated with the MediaPipe library, for obtaining human skeleton key points and hand key points and corresponding data of the human body according to the image; A coordinate processing module, configured to extract the data of the key points of the human skeleton and the key points of the hand, and calculate the corresponding joint angle data and finger bending angle data; A Kalman filter module, configured to smooth the output data of the coordinate processing module; A control signal conversion module, configured to convert the smoothed joint angle data and the finger bending angle data into a servo pulse width and send a corresponding control signal via a serial communication port; and The manipulator module is used to receive the control signal to achieve vision-based human body key point detection and manipulator palm synchronization control.

Images