Industrial robot control method and system based on visual hand tracking

Abstract

The invention provides an industrial robot control method and system based on visual hand tracking, and the method comprises the steps: collecting continuous images through a camera module, transmitting the continuous images to an image processing module, carrying out the various processing of the images through the image processing module, carrying out the recognition of the hand position through key point detection and model reasoning, and carrying out the recognition of the hand position. Hand position difference data such as the horizontal movement distance and the height movement distance are calculated and sent to the data analysis module, and the data analysis module conducts actual distance and pose analysis according to the hand position difference data and sends the analyzed distance and pose data to the industrial robot to control the industrial robot. According to the invention, man-machine interaction control based on visual hand tracking is realized, the hand is tracked through vision, so that the industrial robot is controlled, and the method has the advantages of visual operation, high flexibility, reduction of personnel learning cost and high adaptability.

Description

Technical Field

[0001] This invention relates to the technical field of industrial robot control, and specifically to an industrial robot control method and system based on vision hand tracking. Background Technology

[0002] In the process of industrial automation transforming into intelligent automation, the interaction mode of traditional industrial robots faces fundamental challenges. Existing industrial robots rely on preset programs or teach pendants for control. While they can meet basic operational needs, such as loading and unloading, handling, welding, painting, and grinding, their flexibility and adaptability in human-robot collaboration remain limited. Operators need certain programming skills to write preset programs or indirectly control the robotic arm's movements through complex interfaces, making intuitive and natural interaction impossible and thus posing a challenge for workers operating industrial robots.

[0003] To overcome these limitations, there is an urgent need to develop a real-time control system for industrial robots that can achieve natural interaction. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide an industrial robot control method and system based on visual hand tracking, which realizes natural interaction through visual hand tracking.

[0005] The embodiments of this application are implemented as follows: This application provides an industrial robot control method based on visual hand tracking, characterized by the following steps: Step S1: The camera module captures two consecutive images and transmits them to the image processing module via a signal line; Step S2: The image processing module processes the image captured by the camera module and transmits the hand position difference data to the data analysis module via a signal line; In step S3, the data analysis module performs actual distance and pose analysis based on the hand position difference data, and sends the distance and pose of the actual action required by the industrial robot to the industrial robot for control.

[0006] In some optional implementations, step S2 includes the following specific details: Step S 21 The image processing module uses adaptive thresholding to process the image transmitted from the camera module to obtain the adaptively binarized image. Step S 22 The image processing module performs continuous dilation on Image to obtain Image1; Step S 23The image processing module performs continuous erosion processing on Image1 to obtain Image2; Step S 24 The image processing module performs continuous connected component processing on Image2 to obtain Image3; Step S 25 The image processing module integrates Image, Image1, Image2, and Image3 to perform key point detection and model inference, and identifies the hand position; Step S 26 After the image processing module processes the two consecutive images using the above steps, it compares the hand positions. Specifically, it calculates the horizontal movement distance of the hand by the difference in the pixel values ​​of the hand positions in the two images, and calculates the vertical movement distance of the hand by the change in the size of the hand area in the two images. Step S 27 The image processing module transmits hand position difference data, such as horizontal movement distance and vertical movement distance, to the data analysis module via a signal line.

[0007] In some alternative implementations, step S3 includes the following: Step S 31 The data analysis module performs Kalman filtering on the distance and pose analysis to smooth the data and reduce noise interference. Step S 32 The data analysis module sends the optimal state data, smoothed by Kalman filtering, to the industrial robot for control.

[0008] In some alternative implementations, step S 25 The keypoint detection described herein uses the MediaPipe Hands model, which outputs the coordinates of 21 hand joints. The model inference is based on a lightweight CNN, with the input being the fused features of Image, Image1, Image2, and Image3. Robustness is improved by fusing the results of multi-stage image processing.

[0009] In some alternative implementations, step S 26 The horizontal movement distance of the hand described herein is calculated by the following formula:

[0010] In the formula, These are the coordinates of the center point of the hand in the previous image. The coordinates of the center point of the hand in the next image. This represents the difference in horizontal movement distance of the hand.

[0011] In some alternative implementations, step S26 The height movement distance of the hand described herein is calculated by the following formula:

[0012]

[0013]

[0014] In the formula, (C x C y () represents the coordinates of the screen center point. x2, y2) represents the deviation between the hand and the center point of the screen, and f is the equivalent focal length of the camera. x f is the equivalent focal length of the camera in the horizontal direction (X-axis). y S is the equivalent focal length of the camera in the vertical direction (Y-axis). p S is the actual pixel area of ​​the hand in the camera, and S is the converted center equivalent pixel area of ​​the hand. p' This refers to the area of ​​the hand in the previous image. S represents the change in the size of the hand area, S1 and S2 are the center equivalent pixel areas of the hand after conversion in the previous and subsequent images, respectively, Δz is the height shift distance, and k is the calibration parameter.

[0015] In some alternative implementations, step S 31 The distance and pose analysis described herein is performed using inverse kinematics analysis, as detailed below: For industrial robots, the wrist center point P can be deduced from the hand pose (X, Y, Z) using the following formula. w Coordinates:

[0016]

[0017] In the formula, P e d6 represents the hand position of the end effector, d6 represents the length from the center of the wrist to the end effector, expressed by the length of the 6th pair of sides, and Z represents the unit vector of the Z-axis of the end effector coordinate system in the base coordinate system. Then the angles of the hand joints are solved, where:

[0018]

[0019]

[0020]

[0021]

[0022] In the formula, θ i Let d1 be the rotation angle of the i-th joint, d2 be the offset of the second joint, d1 be the height of the base, and a be the displacement angle of the i-th joint. i Let be the length of the i-th link. This is the transformation matrix from the end effector of the robotic arm to the wrist. Let R be the hand pose matrix. i,j Transformation matrix from wrist to distal end The element in the i-th row and j-th column.

[0023] In some alternative implementations, step S 32 The Kalman filtering process described herein includes the following: definition , representing the six joint angles at time k; Based on the optimal estimated state of the industrial robot joint at the previous moment, predict the prior estimate of the joint angle state at the current moment. and prior estimate of covariance A is the state transition matrix, and Q is the process noise covariance, as detailed below:

[0024]

[0025] In the formula, To reflect the model uncertainty of the k-th joint; Calculate the Kalman gain matrix at the current time step. H is the observation matrix. Since the angle can be directly measured, H = I6. This matrix determines how much confidence should be assigned to the current observation value, i.e., the angle directly calculated from the inverse kinematics; R is the observation noise covariance. , Determined by visual positioning error; Using the observed value z of the current joint angle k The prior estimated state is corrected to obtain the optimal state estimate at the current time after filtering and smoothing. ; Update the covariance estimate corresponding to the optimal estimated state at the current time step. This prepares for the filtering calculation at the next moment; P k The state estimates the covariance matrix, with initial values ​​of... .

[0026] In some alternative implementations, step S 32 The optimal state data mentioned above refers to the optimal state estimate after Kalman filtering smoothing. .

[0027] A lightweight remaining lifetime prediction system based on feature decoupling and sparse optimization, characterized in that it includes: The camera module is used to capture images and is connected to the image processing module via a signal cable; The image processing module is used to process the images captured by the camera module and is connected to the data analysis module via a signal cable; The data analysis module is used to perform actual distance and pose analysis on the hand position difference data in the image, and is connected to the industrial robot via signal lines.

[0028] The beneficial effects of this application are as follows: This application provides an industrial robot control method and system based on vision-based hand tracking. By controlling the industrial robot through vision-based hand tracking, the operational barriers of traditional physical media such as teach pendants and control panels can be eliminated, achieving intuitive control that allows "hands to move as the mind moves." Moreover, operators can complete complex tasks without professional programming knowledge. In harsh industrial environments such as dust, high temperature, and high humidity, using vision-based hand tracking to control the industrial robot and transmitting data wirelessly can effectively avoid personnel injuries. Kalman filtering is used to smooth the data, reducing noise interference and making the movement of the industrial robot more precise and reliable. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart of an industrial robot control method according to an embodiment of this application; Figure 2 This is an image of the operator's hand in an embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0032] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0033] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0034] The features and performance of this application will be further described in detail below with reference to the embodiments.

[0035] The vision-based hand tracking industrial robot control method of the present invention is implemented according to the following technical solution: An industrial robot control system based on vision hand tracking includes a camera module, an image processing module, and a data analysis module.

[0036] Furthermore, in this system, the camera module is used to acquire images, the image processing module is used to process the images acquired by the camera module, and the data analysis module is used to perform actual distance and pose analysis on the hand position difference data in the images.

[0037] Furthermore, in this system, the camera module is connected to the image processing module via a signal line, the image processing module is connected to the data analysis module via a signal line, and the data analysis module is connected to the industrial robot via a signal line.

[0038] The system includes the following steps: Step 1: The camera module captures two consecutive images and transmits them to the image processing module via a signal line.

[0039] Step 2: The image processing module processes the images captured by the camera module and transmits the hand position difference data to the data analysis module via a signal cable. Details are as follows: Step 2.1: The image processing module uses adaptive thresholding to process the image transmitted from the camera module to obtain the adaptively binarized image Image.

[0040] Step 2.2: The image processing module performs continuous dilation on Image to obtain Image1.

[0041] Step 2.3: The image processing module performs continuous erosion processing on Image1 to obtain Image2.

[0042] Step 2.4: The image processing module performs continuous connected component processing on Image2 to obtain Image3.

[0043] Step 2.5: The image processing module integrates Image, Image1, Image2, and Image3 to perform key point detection and model inference to identify the hand position; the key point detection adopts the MediaPipe Hands model, which outputs the coordinates of 21 hand joints; the model inference is based on a lightweight CNN, with the input being the fused features of Image, Image1, Image2, and Image3. By fusing the results of multi-stage image processing, the robustness under harsh environments such as occlusion is improved.

[0044] Step 2.6: After processing the two consecutive images using the above steps, the image processing module compares the hand positions. Specifically, it calculates the horizontal movement distance of the hand based on the pixel difference between the two images and the vertical movement distance based on the change in hand area between the two images. The horizontal movement distance of the hand is calculated using the following formula:

[0045] In the formula, These are the coordinates of the center point of the hand in the previous image. The coordinates of the center point of the hand in the next image. This represents the difference in horizontal movement distance of the hand.

[0046] The distance traveled by altitude is calculated using the following formula:

[0047]

[0048]

[0049] In the formula, (C x C y () represents the coordinates of the screen center point. x2, y2) represents the deviation between the hand and the center point of the screen, and f is the equivalent focal length of the camera. x f is the equivalent focal length of the camera in the horizontal direction (X-axis). y S is the equivalent focal length of the camera in the vertical direction (Y-axis). p S is the actual pixel area of ​​the hand in the camera, and S is the converted center equivalent pixel area of ​​the hand. p' This refers to the area of ​​the hand in the previous image. S represents the change in the size of the hand area, S1 and S2 are the center equivalent pixel areas of the hand after conversion in the previous and subsequent images, respectively, Δz is the height shift distance, and k is the calibration parameter.

[0050] Step 2.7: The image processing module transmits hand position difference data, such as horizontal movement distance and vertical movement distance, to the data analysis module via a signal line.

[0051] Step 3: The data analysis module performs actual distance and pose analysis based on the hand position difference data, and sends the analyzed distance and pose, representing the actual distance and pose required for the industrial robot's movement, to the industrial robot for control. Details are as follows: Step 3.1: The data analysis module performs distance and pose analysis on the data sent by the image processing module. Distance and pose analysis refers to performing inverse kinematics analysis on the industrial robot to achieve distance and pose analysis. Inverse kinematics analysis includes the following: For industrial robots, the wrist center point P can be deduced from the hand pose (X, Y, Z) using the following formula. w Coordinates:

[0052]

[0053] In the formula, P e d6 represents the hand position of the end effector, d6 represents the length from the center of the wrist to the end effector, expressed by the length of the 6th pair of sides, and Z represents the unit vector of the Z-axis of the end effector coordinate system in the base coordinate system. Then the angles of the hand joints are solved, where:

[0054]

[0055]

[0056]

[0057]

[0058] In the formula, θ i Let d1 be the rotation angle of the i-th joint, d2 be the offset of the second joint, d1 be the height of the base, and a be the displacement angle of the i-th joint. i Let be the length of the i-th link. This is the transformation matrix from the end effector of the robotic arm to the wrist. Let R be the hand pose matrix. i,j Transformation matrix from wrist to distal end The element in the i-th row and j-th column.

[0059] Step 3.2: The data analysis module performs Kalman filtering on the distance and pose analysis data to smooth the data and reduce noise interference. Specifically, this includes the following: State modeling: definition , representing the six joint angles at time k.

[0060] State prediction: Based on the optimal estimated state of the joint at the previous moment, predict the prior estimate of the joint angle state at the current moment. and prior estimate of covariance A is the state transition matrix, and Q is the process noise covariance, as detailed below:

[0061]

[0062] In the formula, To reflect the model uncertainty of the k-th joint.

[0063] Kalman gain calculation: Calculate the Kalman gain matrix at the current time step. H is the observation matrix. Since the angle can be directly measured, H = I6. This matrix determines how much confidence should be assigned to the current observation value, i.e., the angle directly calculated from the inverse kinematics; R is the observation noise covariance. , It is determined by visual positioning error.

[0064] State update: using the observed value z of the current joint angle k The prior estimated state is corrected to obtain the optimal state estimate at the current time after filtering and smoothing. .

[0065] Covariance Update: Updates the covariance estimate corresponding to the best estimated state at the current time step. This prepares for the filtering calculation at the next moment; P k The state estimates the covariance matrix, with initial values ​​of... .

[0066] Step 3.3: The data analysis module sends the optimal state data after Kalman filtering smoothing to the industrial robot for control of the industrial robot.

[0067] Example 1 This embodiment is based on a spraying task, where the operator remotely controls the robot to complete the spraying through hand gestures, and the entire process does not require contact with physical equipment.

[0068] Step 1: First, configure the system hardware. The camera module uses a standard industrial camera and is fixed below the hand area. The image processing module uses a K230 chip for real-time visual analysis. The data analysis module uses a CH32V307 chip and connects to a standard six-axis industrial robot.

[0069] Step 2: After the system starts, the camera continuously captures images of the operator's hands (see...). Figure 2 The image is transmitted to the processing module for initial optimization. The processing flow includes adaptive threshold segmentation, converting the image to binary mode. Then, dilation is performed to fill gaps in the hand region, erosion is performed to eliminate minor noise, connected component analysis is performed to identify the complete hand shape, and multiple intermediate images are generated for subsequent fusion.

[0070] Step 3: When the operator moves their hand, the system compares the differences between consecutive frames. The horizontal displacement is calculated using the pixel difference at the center point. x1, y1), vertical displacement is based on the equivalent area of ​​the hand. Derivation of S variation. Area conversion introduces the focal length parameter f. y This involves calibrating distance perception. Specifically, when a hand is close to the camera, its equivalent area relative to the previous frame increases, and vice versa.

[0071] Step 4: The system sends the displacement data to the CH32V307 data analysis module. The data analysis module converts the pixel displacement into actual distance. An inverse kinematics algorithm is used to analyze the robot's joint angles; a forward push of the hand corresponds to the robot's end effector moving forward, and a upward movement of the hand corresponds to the robot's end effector rising higher. Kalman filtering is then used to smooth the joint angle data and reduce jitter. The filtering process is based on state prediction and observation updates, and the final angle values ​​are sent to the robot controller.

[0072] Step 5: After receiving the angle value, the robot controller controls the industrial robot to perform the corresponding actions of the spray gun of the mobile end effector.

Claims

1. An industrial robot control method based on vision hand tracking, characterized in that, Includes the following steps: Step S1: The camera module captures two consecutive images and transmits them to the image processing module via a signal line; Step S2: The image processing module processes the image captured by the camera module and transmits the hand position difference data to the data analysis module via a signal line; In step S3, the data analysis module performs actual distance and pose analysis based on the hand position difference data, and sends the distance and pose of the actual action required by the industrial robot to the industrial robot for control.

2. The industrial robot control method based on vision hand tracking according to claim 1, characterized in that, Step S2 includes the following specific details: Step S 21 The image processing module uses adaptive thresholding to process the image transmitted from the camera module to obtain the adaptively binarized image. Step S 22 The image processing module performs continuous dilation on Image to obtain Image1; Step S 23 The image processing module performs continuous erosion processing on Image1 to obtain Image2; Step S 24 The image processing module performs continuous connected component processing on Image2 to obtain Image3; Step S 25 The image processing module integrates Image, Image1, Image2, and Image3 to perform key point detection and model inference, and identifies the hand position; Step S 26 After the image processing module processes the two consecutive images using the above steps, it compares the hand positions. Specifically, it calculates the horizontal movement distance of the hand by the difference in the pixel values ​​of the hand positions in the two images, and calculates the vertical movement distance of the hand by the change in the size of the hand area in the two images. Step S 27 The image processing module transmits hand position difference data, such as horizontal movement distance and vertical movement distance, to the data analysis module via a signal line.

3. The industrial robot control method based on vision hand tracking according to claim 2, characterized in that, Step S3 includes the following: Step S 31 The data analysis module performs Kalman filtering on the distance and pose analysis to smooth the data and reduce noise interference. Step S 32 The data analysis module sends the optimal state data, smoothed by Kalman filtering, to the industrial robot for control.

4. The industrial robot control method based on vision hand tracking according to claim 2 or 3, characterized in that, Step S 25 The keypoint detection described herein uses the MediaPipe Hands model, which outputs the coordinates of 21 hand joints. The model inference is based on a lightweight CNN, with the input being the fused features of Image, Image1, Image2, and Image3. Robustness is improved by fusing the results of multi-stage image processing.

5. The industrial robot control method based on vision hand tracking according to claim 4, characterized in that, Step S 26 The horizontal movement distance of the hand described herein is calculated by the following formula: In the formula, These are the coordinates of the center point of the hand in the previous image. The coordinates of the center point of the hand in the next image. This represents the difference in horizontal movement distance of the hand.

6. The industrial robot control method based on vision hand tracking according to claim 5, characterized in that, Step S 26 The height movement distance of the hand described herein is calculated by the following formula: In the formula, (C x C y () represents the coordinates of the screen center point. x2, y2) represents the deviation between the hand and the center point of the screen, and f is the equivalent focal length of the camera. x f is the equivalent focal length of the camera in the horizontal direction (X-axis). y S is the equivalent focal length of the camera in the vertical direction (Y-axis). p S is the actual pixel area of ​​the hand in the camera, and S is the converted center equivalent pixel area of ​​the hand. p' This refers to the area of ​​the hand in the previous image. S represents the change in the size of the hand area, S1 and S2 are the center equivalent pixel areas of the hand after conversion in the previous and subsequent images, respectively, Δz is the height shift distance, and k is the calibration parameter.

7. The industrial robot control method based on vision hand tracking according to claim 6, characterized in that, Step S 31 The distance and pose analysis described herein is performed using inverse kinematics analysis, as detailed below: For industrial robots, the wrist center point P can be deduced from the hand pose (X, Y, Z) using the following formula. w Coordinates: In the formula, P e d6 represents the hand position of the end effector, d6 represents the length from the center of the wrist to the end effector, expressed by the length of the 6th pair of sides, and Z represents the unit vector of the Z-axis of the end effector coordinate system in the base coordinate system. Then the angles of the hand joints are solved, where: In the formula, θ i Let d1 be the rotation angle of the i-th joint, d2 be the offset of the second joint, d1 be the height of the base, and a be the displacement angle of the i-th joint. i Let i be the length of the i-th link. This is the transformation matrix from the end effector of the robotic arm to the wrist. Let R be the hand pose matrix. i,j Transformation matrix from wrist to distal end The element in the i-th row and j-th column.

8. The industrial robot control method based on visual hand tracking according to claim 7 The method, characterized in that, Step S 32 The Kalman filtering process described herein includes the following: definition , representing the six joint angles at time k; Based on the optimal estimated state of the industrial robot joint at the previous moment, predict the prior estimate of the joint angle state at the current moment. and prior estimate of covariance A is the state transition matrix, and Q is the process noise covariance, as detailed below: In the formula, To reflect the model uncertainty of the k-th joint; Calculate the Kalman gain matrix at the current time step. H is the observation matrix. Since the angle can be directly measured, H = I6. This matrix determines how much confidence should be assigned to the current observation value, i.e., the angle directly calculated from the inverse kinematics; R is the observation noise covariance. , Determined by visual positioning error; Using the observed value z of the current joint angle k The prior estimated state is corrected to obtain the optimal state estimate at the current time after filtering and smoothing. ; Update the covariance estimate corresponding to the optimal estimated state at the current time step. This prepares for the filtering calculation at the next moment; P k The state estimate covariance matrix is ​​initially set to... 。 9. The industrial robot control method based on visual hand tracking according to claim 8 The method, characterized in that, Step S 32 The optimal state data mentioned above refers to the optimal state estimate after Kalman filtering smoothing. .

10. A lightweight remaining lifetime prediction system based on feature decoupling and sparse optimization, characterized in that, include: The camera module is used to capture images and is connected to the image processing module via a signal cable; The image processing module is used to process the images captured by the camera module and is connected to the data analysis module via a signal cable; The data analysis module is used to perform actual distance and pose analysis on the hand position difference data in the image, and is connected to the industrial robot via signal lines.

Images