A robot teleoperation method and system based on human joint posture

Abstract

This invention belongs to the field of robot control technology and discloses a robot teleoperation method and system based on human joint posture. The method includes: acquiring color and depth images of the operator; extracting the human contour image from the color image and smoothing it to obtain the human contour skeleton, and extracting the first data of human joint points; using a random forest algorithm to obtain the pixel distribution of human body parts through the depth image to generate the second data of human joint points; performing mutual information and weighted fusion on the first and second data of human joint points to obtain target joint data, wherein the target joint includes three joints, and the distance between two joints is adjustable; and using a quaternion calibration method to perform rotation calibration on the target joint data to obtain the pose change of the target joint. This application can realize three-joint control of the end effector, greatly reducing the difficulty of remote operation of the robot and improving the user experience.

Description

Technical Field

[0001] This invention belongs to the field of robot control technology, and more specifically, relates to a robot teleoperation method and system based on human joint posture. Background Technology

[0002] The rapid development of robotics technology has driven innovation and improvement across various industries. However, robot operation in certain specific environments still presents challenges, and the safety, real-time performance, and ease of use in some scenarios have become urgent issues to address. Traditional wired robots connect to the operating equipment via a physical cable of limited length and are controlled through wired signal transmission. Their operating range is relatively small, limiting their application in remote, dangerous, or inaccessible locations, particularly in remote, dangerous, or difficult-to-reach scenarios such as remote experimental teaching, nuclear power plants, industrial production workshops, and disaster relief sites. To address these issues, remote robot operation methods have emerged, providing solutions for remote operation, ensuring operator safety, and expanding the application areas of robots.

[0003] In robot teleoperation methods, the robot and the operator are in different environments. The operator can remotely operate the robot to perform tasks without having to be on-site when the robot is working. Examples include remote teaching in the education field and remote surgery in the medical industry. On the one hand, this provides operators with convenience in terms of time and space; on the other hand, it avoids operators going to dangerous rescue, industrial production, or maintenance sites, ensuring operator safety, reducing operator risks, and providing operators with a safer and more controllable working environment.

[0004] With advancements in computer vision and camera technology, human motion capture technology has become widely used. Some cameras can now capture human activity and posture information, detecting and tracking the skeletal joints of the human body in real time within computer images, thus achieving high-precision human motion capture.

[0005] Human motion information can be translated into control commands for robots, enabling them to perform tasks based on the operator's movements. Using camera-based human motion capture technology to control robots simplifies operation and allows for more intuitive and interactive human-computer interaction. However, current technologies require coupling between human joints and robot control parts; for example, coupling a human hand joint to the robot's end effector. This is inconvenient for operators with hand disabilities, and existing technologies require the coordination of multiple joints to complete a specified action, necessitating redundant motions or equipment, resulting in significant control challenges. Summary of the Invention

[0006] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a robot teleoperation method and system based on human joint posture, which can realize three-joint control of the end effector, greatly reducing the difficulty of remote operation of the robot and improving the user experience.

[0007] To achieve the above objectives, according to one aspect of the present invention, a robot teleoperation method based on human joint posture is provided, comprising: S1: acquiring a color image and a depth image of the operator; S2: extracting a human contour image from the color image and performing smoothing processing to obtain a human contour skeleton and extracting first data of human joint points; and using a random forest algorithm to obtain the pixel distribution of human body parts through the depth image to generate second data of human joint points; performing mutual information and weighted fusion on the first data of human joint points and the second data of human joint points to obtain target joint data, wherein the target joint includes three joints, of which the distance between two joints is adjustable; S3: using a quaternion calibration method to perform rotation calibration on the target joint data to obtain the pose change of the target joint, wherein one joint controls the displacement of the robot end effector, and the other two joints control the movement of the end effector gripper.

[0008] Preferably, step S3 further includes judging the pose change of the target joint. If the pose change is greater than a preset threshold, the maximum range of the threshold is taken as the pose change.

[0009] Preferably, the specific steps of extracting the human body contour image from the color image and performing smoothing processing to obtain the human body contour skeleton in step S2 are as follows: using a human body segmentation method to segment the operator's human body from the color image to obtain the operator's human body contour; performing smoothing processing on the human body contour to obtain a binary image of the human body contour; and using a thinning algorithm based on the binary image to obtain the human body contour skeleton.

[0010] Preferably, step S2 involves using a random forest algorithm to obtain the pixel distribution of human body parts from a depth image. Specifically, obtaining human body parts involves using a random forest algorithm to obtain the pixel distribution of body parts from a depth image and then performing cross-pixel merging to obtain the human body parts.

[0011] Preferably, step S3 further includes filtering and filling the data after rotation calibration.

[0012] This application also provides a robot teleoperation system based on human joint posture, comprising: an image acquisition module for acquiring color and depth images of the operator; an image processing module for extracting human contour images from the color images and smoothing them to obtain a human contour skeleton, and extracting first data of human joint points; and using a random forest algorithm to obtain the pixel distribution of human body parts through the depth image to generate second data of human joint points; performing mutual information and weighted fusion on the first and second data of human joint points to obtain target joint data, wherein the target joint includes three joints, two of which are adjustable in distance; and using a quaternion calibration method to perform rotation calibration on the target joint data to obtain the pose change of the target joint, wherein one joint controls the displacement of the robot end effector, and the other two joints control the action of the end effector gripper; and a control module for controlling the movement of the end effector based on the pose change of the target joint.

[0013] Preferably, the control module further includes: judging the pose change of the target joint; if the pose change is greater than a preset threshold, then the maximum range of the threshold is taken as the pose change.

[0014] Preferably, the system further includes a monitoring module and a display module: the monitoring module is used to monitor the robot's operating status in real time; the display module is used to display the movement of the robot and the virtual robot in real time.

[0015] In summary, compared with the prior art, the robot teleoperation method and system based on human joint posture provided by the present invention mainly have the following advantages:

[0016] Beneficial effects:

[0017] 1. By weighted fusion of color and depth images, and by assigning weights to the localization results of different image sources, the reliability of these weights influences the final joint pose calculation. This allows for the enhancement of information from one image source based on specific needs, resulting in more accurate joint localization and improved robot control precision. Remote operation of the robot can be achieved with only three joints, without requiring the operator to specify which joints to operate or perform complex limb movements. This further reduces the difficulty of robot operation and provides a more flexible tool for robot operators in various fields.

[0018] 2. This invention uses quaternion calibration technology to correct deviations caused by camera installation, ensuring that the camera can still accurately capture image information within a certain angular error range. The camera installation position does not need to be strictly directly facing the camera; a certain angular error is permissible, reducing the difficulty of camera installation and improving user convenience.

[0019] 3. Compared with traditional feedback methods, this invention displays both the robot's operation and the virtual robot's operation in real time on a screen. This dual information feedback allows operators to have a more comprehensive understanding of the robot's status and environment, helping to identify potential problems in advance and thus make better decisions and execute the next steps.

[0020] 4. This invention is based on a camera-controlled remote robot and designs a robot teleoperation method based on human joint posture. Operators can remotely control the robot to perform various tasks, improving operational convenience and ensuring operator safety. This invention has wide applications, including scientific laboratories, industrial production, and medical surgery. In remote education, experiments, and surgeries, it provides users with spatial and temporal convenience; in industrial production, it significantly reduces operator risks, providing a safer and more controllable working environment. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the steps of the robot teleoperation method based on human joint posture according to the present invention.

[0022] Figure 2 This is a flowchart illustrating the robot teleoperation method based on human joint posture according to the present invention.

[0023] Figure 3 This is a block diagram of the robot teleoperation system based on human joint posture according to the present invention;

[0024] Figure 4 This is a flowchart of the present invention for obtaining human joint positioning through images;

[0025] Figure 5 This is a schematic diagram of the hardware connection of the robot teleoperation system based on human joint posture according to the present invention;

[0026] Figure 6 This is a schematic diagram of the human hand joints according to an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0028] This invention provides a robot teleoperation method based on human joint posture, such as... Figures 1-3 As shown, the method includes the following steps S1 to S3.

[0029] S1: Acquire color and depth images of the operator.

[0030] A camera can be used to capture color and depth images of the operator in real time, and the captured image data can be transmitted to the host computer in real time.

[0031] S2: Extract the human body contour image from the color image and smooth it to obtain the human body contour skeleton, and extract the first data of human body joints; and use the random forest algorithm to obtain the pixel distribution of human body parts through the depth image to generate the second data of human body joints; perform mutual information and weighted fusion on the first data of human body joints and the second data of human body joints to obtain the target joint data, wherein the target joint includes three joints, two of which are adjustable in distance, and one joint controls the overall movement of the end effector, and the other two joints control the opening and closing of the end effector.

[0032] The specific steps for extracting the human body contour image from the color image and smoothing it to obtain the human body contour skeleton are as follows:

[0033] The human body of the operator is segmented from the color image using a human body segmentation method to obtain the outline of the operator's human body;

[0034] The human body contour is smoothed to obtain a binary image of the human body contour.

[0035] The human body outline skeleton is obtained by using a thinning algorithm based on the binary image.

[0036] The random forest algorithm is used to obtain the pixel distribution of human body parts from depth images. Specifically, the human body parts are obtained as follows:

[0037] The random forest algorithm is used to obtain the distribution of body parts by pixels in the depth image, and cross-pixel merging is performed to obtain the human body parts.

[0038] The human body parts are weighted and fused with the human body contour skeleton, and the target joints are located. The specific weighted fusion steps are as follows:

[0039] Ensure that the human joint data obtained in both methods are in the same format and coordinate system. If there are inconsistencies, they need to be converted to be in the same coordinate system.

[0040] For the first and second sets of human joint data, calculate the mutual information between them, and use the result of the mutual information calculation as a reference for weighting to assign weights to the two sets of joint data.

[0041] The first and second sets of human body joint data are weighted and fused using predetermined weights. The fusion formula is as follows:

[0042] F(x,y)=w1·F1(x,y)+w2·F2(x,y)

[0043] Where F is the fused result, F1 and F2 are two sets of joint data, and w1 and w2 are the corresponding weights.

[0044] Target joint data is generated, and the fused result is the final target joint data.

[0045] S3: The target joint data is rotated and calibrated using the quaternion calibration method to obtain the pose change of the target joint, where one joint controls the displacement of the robot end effector and the other two joints control the movement of the end effector gripper.

[0046] The quaternion calibration method was used to perform rotational calibration on the collected human joint data.

[0047] In a further preferred embodiment, the calibrated data is subjected to noise filtering through image filtering, and missing data is filled in.

[0048] Human joint posture changes are converted into end effector pose changes, and robot control is achieved through the pose changes of three human joints. One human joint controls the end effector's movement, and the pose change information of this joint is used to obtain the adjusted pose data of the end effector. The other two human joints control the end effector's task actions, and the pose changes of these two joints are used to obtain the action command information of the end effector gripper.

[0049] In a further preferred embodiment, step S3 further includes judging the pose change of the target joint. If the pose change is greater than a preset threshold, the maximum range of the threshold is taken as the pose change.

[0050] Another aspect of this application provides a robot teleoperation system based on human joint posture, such as... Figure 4 and Figure 5 As shown, the system includes an image acquisition module, an image processing module, and a control module.

[0051] The image acquisition module is used to acquire color and depth images of the operator. The module includes a camera positioned in front of the operator. As the operator moves their limbs, the camera acquires color and depth images of the operator. The acquired image information is then transmitted to the image processing module.

[0052] The image processing module includes a host computer. Images are processed by the host computer. For color images, the operator's body is segmented using a human body segmentation method to obtain the operator's human body contour image. The image is then smoothed to obtain a smooth and connected binary image of the human body contour. A thinning algorithm is used to extract the human body contour skeleton, locate the human body joints, and obtain the first data of human body joint points. For depth images, a random forest algorithm is used to infer the distribution of body parts of pixels using the depth image, and cross-pixel merging is performed to generate the second data of human body joint points. Using the two methods described above, mutual information and fusion weighting processing are performed on the located first and second data of human body joint points to finally obtain the human body joint pose.

[0053] Furthermore, the image processing module uses quaternions to perform rotation calibration on the data. Specifically, the operator moves along the front-back direction and sequentially collects the coordinates of two points P. a (x1, y1, z1) and P b (x2, y2, x2), by P a and P b The vector before calibration is l1 = (x2-x1, y2-y1, z2-z1). The length of the calibrated vector along the x-axis is the magnitude of l1, and the lengths along the y-axis and z-axis are 0. Therefore, l2 = (||l1||2, 0, 0). The rotation quaternion q is obtained using the method described above.

[0054]

[0055] Where i, j, and k are the normal vectors of the quaternion, α, β, and γ are the Euler angles corresponding to the quaternion, θ is the angle between vectors l1 and l2, and vector n is the normal vector perpendicular to l1 and l2.

[0056] Furthermore, for any vector l = (p) in the space captured by the camera... x p y p z The vector is calibrated using the quaternion q, and the calibrated result is as follows:

[0057]

[0058] In the formula, l r The vector is calibrated; q is the rotation quaternion. This represents quaternion multiplication.

[0059] By correcting the deviation caused by equipment installation using the above method, and because quaternion calibration has been performed, the position of the camera does not need to be strictly required to be perfectly aligned during installation, and a certain installation angle error is allowed.

[0060] The calibrated data is filtered to remove noise through image filtering, such as Gaussian filtering, mean filtering, and median filtering. These are just examples and are not limited to the methods used to filter noise from images.

[0061] Furthermore, missing data can be filled in using methods such as interpolation and content-based image completion. These are just examples and are not limited to methods used to fill in missing data.

[0062] When an operator performs an action, there is an initial posture. As the operator performs a certain action, the posture of the joints in the human body changes. Referring to the initial posture, the position and posture changes of the joints can be calculated according to the process described above.

[0063] Among human joints, the hand joints are the most flexible, allowing for easy movement and rotation compared to other joints. Human hand joints can be used to control robots, utilizing joints such as the wrist, fingertips, and thumb. Figure 6 As shown. Specifically, the hand joints are mapped to the movements of the robot's end effector, and the position and orientation of the robot's end effector are controlled by the wrist joint. The movements of the robot's end effector are controlled by the fingertips and thumb, and the distance between the fingertips and thumb is mapped to the distance between the opening of the end effector grippers, controlling the opening or closing of the grippers.

[0064] The use of hand joints to control the robot is merely an example; the present invention is not limited to remotely operating the robot using hand joint postures.

[0065] The image, after processing to obtain pose information, is transmitted via network to a computer in the robot's working environment using a host computer.

[0066] The control module includes a computer and a robot. The computer uses the transmitted pose information to calculate the pose of the end effector of the robotic arm. Based on the inverse kinematics algorithm, it plans the trajectory of the end effector, solves the rotation angle of each joint of the robotic arm, and drives the joint motors to move the end effector to the corresponding position and adjust its posture. The end effector opens or closes to perform the task.

[0067] In actual operation, the coordinate range that the robot's end effector can reach is smaller than the coordinate range of the camera. In order to prevent the coordinates of the end effector after human posture mapping from exceeding the robot's execution range, the control module will judge the pose of the end effector: if the mapped robot end effector coordinates exceed the maximum execution range, the maximum offset within the reachable range will be used as the input data.

[0068] The control module calculates the opening distance of the end effector grippers based on the offset of the thumb and fingertip joints. The maximum opening distance of the end effector grippers is D. This system will judge the mapped opening distance: if the mapped opening distance of the robot end effector grippers exceeds the maximum distance D, the maximum distance D will be used as the input data. The minimum closing degree makes the clamping distance between the two grippers zero. If the change in the closing distance of the mapped robot end effector grippers exceeds the current state distance, the end effector gripper distance will be restored to zero, or the clamping force of the end effector grippers will reach a certain value.

[0069] The robot control module receives data transmitted from the host computer, including the rotation angle information of each joint and the instruction information of the end effector gripper. Upon receiving the instruction data, the robot drives the motors to rotate each joint by a certain angle, adjusting the posture of the end effector, causing the end effector gripper to open or close.

[0070] When the robot is running, it transmits its own motion data to the computer.

[0071] This application system also includes a monitoring module and a display module.

[0072] The monitoring module is used to monitor the robot's operating status in real time. The monitoring camera captures images of the robot in real time and transmits the captured images to the computer. The computer then transmits the captured image information and the robot's motion data to the host computer via a network.

[0073] The host computer uses the received robot motion data to drive the virtual robot to move and displays the image to the operator through the display module.

[0074] The display module includes a display screen that shows both real-time and virtual robot operation. The display screen is placed in front of the operator, who views the images on the screen to assess the effectiveness of human posture control of the robot.

[0075] The image acquisition module, image processing module, and display module are located in a remote operating environment.

[0076] The control module and monitoring module are located in the robot's working environment.

[0077] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A robot teleoperation method based on human joint posture, characterized in that, include: S1: Acquire color and depth images of the operator; S2: Extract the human body contour image from the color image and smooth it to obtain the human body contour skeleton, and extract the first data of human body joints; and use the random forest algorithm to obtain the pixel distribution of human body parts through the depth image to generate the second data of human body joints. The first and second data of human joint points are mutually fused and weighted to obtain the target joint data. The target joint includes three joints, two of which have an adjustable distance. S3: The target joint data is rotated and calibrated using the quaternion calibration method to obtain the pose change of the target joint, where one joint controls the displacement of the robot end effector and the other two joints control the movement of the end effector gripper.

2. The robot teleoperation method according to claim 1, characterized in that, Step S3 further includes judging the pose change of the target joint. If the pose change is greater than a preset threshold, the maximum range of the threshold is taken as the pose change.

3. The robot teleoperation method according to claim 1 or 2, characterized in that, The specific steps for extracting the human body contour image from the color image and performing smoothing processing to obtain the human body contour skeleton in step S2 are as follows: The human body of the operator is segmented from the color image using a human body segmentation method to obtain the outline of the operator's human body; The human body contour is smoothed to obtain a binary image of the human body contour. The human body outline skeleton is obtained by using a thinning algorithm based on the binary image.

4. The robot teleoperation method according to claim 1 or 2, characterized in that, Step S2 describes using the random forest algorithm to obtain the pixel distribution of human body parts from the depth image. Specifically, obtaining the human body parts involves: The random forest algorithm is used to obtain the distribution of body parts by pixels in the depth image, and cross-pixel merging is performed to obtain the human body parts.

5. The robot teleoperation method according to claim 1, characterized in that, Step S3 also includes filtering and filling the data after rotation calibration.

6. A robot teleoperation system based on human joint posture, characterized in that, include: Image acquisition module: used to acquire color and depth images of the operator; Image processing module: used to extract human body contour images from color images and perform smoothing processing to obtain the human body contour skeleton, extract the first data of human body joints; and use the random forest algorithm to obtain the pixel distribution of human body parts through depth images to generate the second data of human body joints. The first and second human joint data are mutually fused and weighted to obtain target joint data, wherein the target joint includes three joints, two of which have an adjustable distance; and the target joint data is rotated and calibrated using a quaternion calibration method to obtain the pose change of the target joint, wherein one joint controls the displacement of the robot end effector, and the other two joints control the action of the end effector gripper. Control module: Used to control the movement of the end effector based on the pose changes of the target joint.

7. The robot teleoperation system according to claim 6, characterized in that, The control module also includes: The pose change of the target joint is judged. If the pose change is greater than a preset threshold, the maximum range of the threshold is taken as the pose change.

8. The robot teleoperation system according to claim 6 or 7, characterized in that, It also includes a monitoring module and a display module: The monitoring module is used to monitor the robot's operating status in real time; The display module is used to display the movement of the robot and the virtual robot in real time.

Images