Robotic teleoperation method and system based on multi-view confidence fusion

Abstract

The application provides a multi-view confidence fusion-based arm robot remote operation method and system, and relates to the technical field of robots; the application fully scores and distinguiates relevant information displayed by multiple different views, determines the confidence of different views, and thus can better select among multiple camera views, thereby relieving problems caused by hand self-occlusion, overcoming the problem of hand self-occlusion in dexterous operation, and further improving the robustness, accuracy and efficiency of visual-based remote control operation while maintaining low cost and high adaptability, thereby providing strong support for dexterous operation research based on imitation learning.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a teleoperation method and system for an arm-handed robot based on multi-view confidence fusion. Background Technology

[0002] The rapid development of robotics technology has led to impressive results in the dexterity of robotic arms, overcoming the control difficulties caused by high degrees of freedom and enabling operators to solve various complex tasks. It also provides an effective and intuitive way to collect robot demonstration data. Especially with recent significant advancements in demonstration-based learning methods, teleoperation has gained considerable attention. Traditional teleoperation is primarily used in environments inaccessible to humans, such as space exploration, disaster relief, and telemedicine. Recent research has shifted its focus to collecting demonstration data through teleoperation for policy learning to achieve autonomous task completion. For teleoperation of robotic arms with dexterous hands, due to the high degree of freedom and the differences between the robotic arm and human anatomy, existing methods employ VR devices, exoskeletons, and motion capture devices. These methods typically face challenges such as operator inaccessibility and high material costs. However, traditional vision-based teleoperation systems are susceptible to hand occlusion and environmental factors, reducing operational accuracy and failing to meet the requirements of convenience, speed, and economy.

[0003] Therefore, it is necessary to provide a teleoperation method and system for arm-handed robots based on multi-view confidence fusion to solve the above-mentioned technical problems. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a teleoperation method and system for an arm-hand robot based on multi-view confidence fusion, which solves the problem of reduced operational accuracy caused by hand self-occlusion and environmental factors in the prior art.

[0005] The teleoperation method for an arm-handed robot based on multi-view confidence fusion provided by this invention includes the following steps: S1. Simultaneously capture images of the operator's hands from different perspectives using multiple cameras to form multi-view video stream data; S2. Obtain wrist key points and finger root key points from multi-view video stream data, construct 6D pose after mixing and filtering, and solve the joint motion angles of the robotic arm through an inverse kinematics solver. S3. Calculate the overall confidence score for all views in all multi-view video stream data. The overall confidence score is obtained by calculating the visibility score, image sharpness score and temporal stability score of all views and then using a weighted summation method. S4. Select the viewpoint image with the highest confidence score as the main viewpoint, extract the coordinates of the key points of the hand, and calculate the target angle of each joint of the robot hand by constructing an optimization objective function to minimize the difference between the operator's hand and the mechanical fingertip vector. S5. The target angles of each joint of the robotic arm and the joint motion angles of the robotic arm are synchronously sent to the simulation or physical robot platform to drive it to complete the operation task.

[0006] Preferably, step S1 specifically includes: by configuring at least one RGB-D camera and at least one RGB camera, synchronously acquiring images of the operator's hand from different perspectives, and aligning the hand images from multiple perspectives according to the timeline to form multi-view video stream data.

[0007] Preferably, the specific steps of step S2 are as follows: S201. The wrist key point and finger root key point data acquired by the RGB-D camera in the multi-view video stream data are mixed and filtered. The mixed filtering specifically includes spatial median filtering, outlier removal and pre-value filling, Kalman filtering and moving average filtering. S202. Map the wrist key points after hybrid filtering to the target position at the end of the robotic arm; S203. Based on the wrist key points and the key points at the base of the index finger and the base of the little finger in the mixed-filtered wrist key points and finger root key points data, calculate the local coordinate axes and map them to the target posture at the end of the robotic arm. S204. Based on the target position and target pose combination of the robotic arm end effector, a 6D pose is obtained, and the joint motion angles of the robotic arm are obtained by solving the 6D pose inverse kinematics solver.

[0008] Preferably, in step S3, the visibility score includes: obtaining the currently visible hand key points from multi-view video stream data, and calculating the ratio of the convex hull area formed by the currently visible hand key points to the theoretical complete hand area, used to evaluate the degree of occlusion; the image sharpness score includes: obtaining the ratio of the image Laplacian variance to the preset maximum variance, used to evaluate the degree of image blur; the temporal stability score includes: calculating the normalized value of the average displacement of the hand key points between the current frame and the previous frame, wherein the smaller the displacement, the higher the stability.

[0009] Preferably, in step S4, the calculation formula for the objective function specifically includes: ; In the formula, This indicates the joint position during time t in the movement control process of a dexterous hand. This represents the vector of the i-th key point of the hand calculated based on the detected finger key points. It is the i-th positive kinematics function, with the joint positions of the robot hand as the output. It is a scaling factor used to account for differences in hand size; It is a penalty item used to improve time smoothness.

[0010] The process of selecting the viewpoint image with the highest confidence score as the main viewpoint and extracting the coordinates of its hand key points specifically includes using the MediaPipe algorithm to extract the 2.5D coordinates of 21 hand key points.

[0011] This invention also provides a teleoperation system for an arm-handed robot based on multi-view confidence fusion, comprising: The data acquisition module is used to simultaneously acquire images of the operator's hands from different perspectives using multiple cameras, forming multi-view video stream data; The inverse kinematics solving module is used to obtain wrist key points and finger root key points from multi-view video stream data, construct 6D pose after mixing and filtering, and solve the joint motion angles of the robotic arm through the inverse kinematics solver. The confidence assessment module is used to calculate the overall confidence score of all views in all multi-view video stream data. The overall confidence score is obtained by calculating the visibility score, image sharpness score and temporal stability score of all views and then using a weighted summation method. The hand motion redirection module selects the viewpoint image with the highest confidence score as the main viewpoint, extracts the coordinates of the key points of the hand, and calculates the target angles of each joint of the robotic hand by constructing an optimization objective function to minimize the difference between the operator's hand and the robotic fingertip vectors. The execution control module is used to synchronously send the target angles of each joint of the robotic arm and the joint motion angles of the robotic arm to the simulation or physical robot platform, driving it to complete the operation task.

[0012] The present invention also provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the described teleoperation method for a hand-arm robot based on multi-view confidence fusion.

[0013] The present invention also provides a computer-readable storage medium storing computer instructions for causing a processor to execute the described teleoperation method for an arm-handed robot based on multi-view confidence fusion.

[0014] The present invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the teleoperation method for an arm-handed robot based on multi-view confidence fusion.

[0015] Compared with related technologies, the teleoperation method and system for arm-handed robots based on multi-view confidence fusion provided by this invention have the following beneficial effects: This invention fully scores and judges relevant information displayed from multiple different perspectives, determines the confidence level of different perspectives, and thus makes better selection among multiple camera perspectives. This alleviates the problem caused by hand self-occlusion and overcomes the problem of hand self-occlusion in dexterity operation. In this way, while maintaining low cost and high adaptability, it improves the robustness, accuracy and efficiency of vision-based teleoperation, and provides strong support for the research of dexterity operation based on imitation learning. Attached Figure Description

[0016] Figure 1 The flowchart shows the teleoperation method for an arm-handed robot based on multi-view confidence fusion according to the present invention. Figure 2 This is a system block diagram of the arm-handed robot teleoperation system based on multi-view confidence fusion according to the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0018] Although the steps in this invention are arranged by reference numerals, this is not intended to limit the order of the steps. Unless the order of the steps is explicitly stated or the execution of a step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items.

[0019] Example 1 like Figure 1 As shown, the teleoperation method for a hand-arm robot based on multi-view confidence fusion includes the following steps: S1. Simultaneously capture images of the operator's hands from different perspectives using multiple cameras to form multi-view video stream data.

[0020] Specifically, by configuring at least one RGB-D camera and at least one RGB camera, images of the operator's hands are simultaneously acquired from different perspectives, and the hand images from multiple perspectives are aligned along the timeline and combined to form multi-view video stream data.

[0021] S2. Obtain wrist key points and finger root key points from multi-view video stream data, construct 6D pose after mixing and filtering, and solve the joint motion angles of the robotic arm through an inverse kinematics solver.

[0022] In the specific implementation process, the specific steps of step S2 are as follows: S201. The wrist key point and finger root key point data acquired by the RGB-D camera in the multi-view video stream data are mixed and filtered. The mixed filtering specifically includes spatial median filtering, outlier removal and pre-value filling, Kalman filtering and moving average filtering.

[0023] S202. Map the wrist key points after hybrid filtering to the target position at the end of the robotic arm.

[0024] S203. Based on the wrist key points and the key points at the base of the index finger and the base of the little finger in the mixed-filtered wrist key points and finger root key points data, calculate the local coordinate axes and map them to the target posture at the end of the robotic arm.

[0025] S204. Based on the target position and target pose combination of the robotic arm end effector, a 6D pose is obtained, and the joint motion angles of the robotic arm are obtained by solving the 6D pose inverse kinematics solver.

[0026] Specifically, the 6D pose of the wrist is obtained and mapped onto the end effector of the robotic arm to define the wrist position information. and wrist rotation information The wrist key point location information is transformed into 3D coordinates from the 2.5D wrist key points identified by Mediapipe through a hybrid filtering process. .

[0027] Use the same procedure to obtain the base of the index finger. and the base of the little finger The spatial coordinates are obtained by normalizing the vectors from the wrist key point to the key point at the base of the index finger and to the key point at the base of the little finger, and using them as two axes. and Construct a plane parallel to the palm. The specific calculation formula can be expressed as follows: ; in, This indicates a normalization operation.

[0028] The third axis perpendicular to the plane is obtained through the cross product. The specific calculation formula can be expressed as: ; in, This represents the cross product operation.

[0029] Rotation at key wrist points obtained through orthogonal combination , can be represented as: ; And combined with the motion vector to determine the complete 6D wrist pose. , can be represented as: ; The human wrist posture Mapped to the end effector of the robotic arm, serving as the target pose of the end effector. The change of this coordinate over time is mapped to the change in the direction of the end effector. Then, the inverse kinematics of the arm is solved using mink, a Python differential inverse kinematics library based on the MuJoCo physics engine and the Pinocchio inverse kinematics library. It can generate smooth and feasible motions for the robotic arm and finally obtain the joint angles of the robotic arm.

[0030] S3. Calculate the overall confidence score for all views in all multi-view video stream data. The overall confidence score is obtained by calculating the visibility score, image sharpness score and temporal stability score of all views and then using a weighted summation method.

[0031] Specifically, the formula for calculating the visibility score includes: ; ; In the formula, This represents the area of ​​the currently visible hand, specifically the area of ​​the polygon enclosed by the currently visible hand keypoints. The number of polygon vertices involved in the area calculation is 21 in this embodiment. These are the location information of key hand points identified by Mediapipe. This is an estimate of the palm width, mainly measured from the base of the index finger and little finger. and The estimation is based on the distance between key points. It's an estimate of the hand length, calculated from the wrist and the base of the middle finger. and The estimation is based on the distance between key points. It is an estimated complete theoretical area of ​​the hand, where and The width and length of the image are 640 and 480, respectively.

[0032] Image sharpness score, the formula for which is calculated includes: ; In the formula, It's a perspective The variance of the image Laplacian operator, Used for normalization, the values ​​are determined based on the parameters of cameras with the same viewpoint. In this embodiment, the values ​​for the two viewpoints used are 200 and 400, respectively. The clearer the image, the better. The closer the score is to 1, the more reliable the pose. This is used to evaluate the impact of image sharpness on image credibility from different viewpoints.

[0033] The time stability score is calculated using the following formula: ; In the formula, This sets the key points on the wrist in the current frame and the previous frame. It is the average displacement where It is a scaling parameter that controls the influence of displacement confidence. Small displacements (close to 1) indicate stable results.

[0034] The weighted summation method is specifically calculated using the following formula: ; In the formula, , , All are weighting coefficients and In this embodiment, the weighting coefficient is set. , , .

[0035] S4. Select the viewpoint image with the highest confidence score as the main viewpoint, extract the coordinates of the key points of the hand, and calculate the target angle of each joint of the robot hand by constructing an optimization objective function to minimize the difference between the operator's hand and the mechanical fingertip vector.

[0036] Specifically, based on the coordinates of 21 key hand points in the corresponding image, the vector between the fingertips of the operator's hand is calculated, and its norm is calculated with a vector between the fingertips of the robotic hand. The optimal solution is then obtained through optimization to determine the angle of the robotic hand's motor joints. The specific calculation formula is as follows: ; In the formula, This indicates the joint position during time t in the movement control process of a dexterous hand. This represents the vector of the i-th key point of the hand calculated based on the detected finger key points. It is the i-th positive kinematics function, with the joint positions of the robot hand as the output. It is a scaling factor used to account for differences in hand size; It is a penalty item used to improve time smoothness; The viewpoint image with the highest overall confidence score was selected as the main viewpoint, and the coordinates of its hand key points were extracted. Specifically, the 2.5D coordinates of 21 hand key points were extracted using the MediaPipe algorithm. The MediaPipe algorithm is a lightweight hand detection tool based on RGB that processes multi-view images and can run in real time on the CPU. It is used to detect 21 hand key points in the wrist frame and obtain the 2.5D coordinates of all 21 key points. The hand key point numbers are represented by 0, 1, k, and 20.

[0037] S5. The target angles of each joint of the robotic arm and the joint motion angles of the robotic arm are synchronously sent to the simulation or physical robot platform to drive it to complete the operation task.

[0038] Example 2 like Figure 2 As shown, the arm-handed robot teleoperation system based on multi-view confidence fusion includes: The data acquisition module is used to simultaneously acquire images of the operator's hands from different perspectives using multiple cameras, forming multi-view video stream data; The inverse kinematics solving module is used to obtain wrist key points and finger root key points from multi-view video stream data, construct 6D pose after mixing and filtering, and solve the joint motion angles of the robotic arm through the inverse kinematics solver. The confidence assessment module is used to calculate the overall confidence score of all views in all multi-view video stream data. The overall confidence score is obtained by calculating the visibility score, image sharpness score and temporal stability score of all views and then using a weighted summation method. The hand motion redirection module selects the viewpoint image with the highest confidence score as the main viewpoint, extracts the coordinates of the key points of the hand, and calculates the target angles of each joint of the robotic hand by constructing an optimization objective function to minimize the difference between the operator's hand and the robotic fingertip vectors. The execution control module is used to synchronously send the target angles of each joint of the robotic arm and the joint motion angles of the robotic arm to the simulation or physical robot platform, driving it to complete the operation task.

[0039] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0040] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0041] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

Claims

1. A teleoperation method for a hand-arm robot based on multi-view confidence fusion, characterized in that, Includes the following steps: S1. Simultaneously capture images of the operator's hands from different perspectives using multiple cameras to form multi-view video stream data; S2. Obtain wrist key points and finger root key points from multi-view video stream data, construct 6D pose after mixing and filtering, and solve the joint motion angles of the robotic arm through an inverse kinematics solver. S3. Calculate the overall confidence score for all views in all multi-view video stream data. The overall confidence score is obtained by calculating the visibility score, image sharpness score and temporal stability score of all views and then using a weighted summation method. S4. Select the viewpoint image with the highest confidence score as the main viewpoint, extract the coordinates of the key points of the hand, and calculate the target angle of each joint of the robot hand by constructing an optimization objective function to minimize the difference between the operator's hand and the mechanical fingertip vector. S5. The target angles of each joint of the robotic arm and the joint motion angles of the robotic arm are synchronously sent to the simulation or physical robot platform to drive it to complete the operation task.

2. The teleoperation method for a hand-arm robot based on multi-view confidence fusion according to claim 1, characterized in that, Step S1 specifically includes: by configuring at least one RGB-D camera and at least one RGB camera, synchronously acquiring images of the operator's hand from different perspectives, aligning the hand images from multiple perspectives according to the timeline, and combining them to form multi-view video stream data.

3. The teleoperation method for a hand-arm robot based on multi-view confidence fusion according to claim 1, characterized in that, The specific steps of step S2 are as follows: S201. The wrist key point and finger root key point data acquired by the RGB-D camera in the multi-view video stream data are mixed and filtered. The mixed filtering specifically includes spatial median filtering, outlier removal and pre-value filling, Kalman filtering and moving average filtering. S202. Map the wrist key points after hybrid filtering to the target position at the end of the robotic arm; S203. Based on the wrist key points and the key points at the base of the index finger and the base of the little finger in the mixed-filtered wrist key points and finger root key points data, calculate the local coordinate axes and map them to the target posture at the end of the robotic arm. S204. Based on the target position and target pose combination of the robotic arm end effector, a 6D pose is obtained, and the joint motion angles of the robotic arm are obtained by solving the 6D pose inverse kinematics solver.

4. The teleoperation method for a hand-arm robot based on multi-view confidence fusion according to claim 1, characterized in that, In step S3, the visibility score includes: obtaining the currently visible hand key points from the multi-view video stream data, and calculating the ratio of the convex hull area formed by the currently visible hand key points to the theoretical complete hand area, which is used to evaluate the degree of occlusion; the image sharpness score includes: obtaining the ratio of the image Laplacian variance to the preset maximum variance, which is used to evaluate the degree of image blur; the temporal stability score includes: calculating the normalized value of the average displacement of the hand key points between the current frame and the previous frame, where a smaller displacement indicates higher stability.

5. The teleoperation method for a hand-arm robot based on multi-view confidence fusion according to claim 1, characterized in that, In step S4, the specific formula for calculating the objective function includes: ； In the formula, This indicates the joint position during time t in the movement control process of a dexterous hand. This represents the vector of the i-th key point of the hand calculated based on the detected finger key points. It is the i-th positive kinematics function, with the joint positions of the robot hand as the output. It is a scaling factor used to account for differences in hand size; It is a penalty item used to improve time smoothness.

6. The teleoperation method for a robotic arm based on multi-view confidence fusion according to claim 1, characterized in that, In step S4, selecting the viewpoint image with the highest confidence score as the main viewpoint and extracting the coordinates of its hand key points specifically includes using the MediaPipe algorithm to extract the 2.5D coordinates of 21 hand key points.

7. A teleoperation system for a hand-arm robot based on multi-view confidence fusion, employing the teleoperation method for a hand-arm robot based on multi-view confidence fusion as described in any one of claims 1-6, characterized in that, include: The data acquisition module is used to simultaneously acquire images of the operator's hands from different perspectives using multiple cameras, forming multi-view video stream data; The inverse kinematics solving module is used to obtain wrist key points and finger root key points from multi-view video stream data, construct 6D pose after mixing and filtering, and solve the joint motion angles of the robotic arm through the inverse kinematics solver. The confidence assessment module is used to calculate the overall confidence score of all views in all multi-view video stream data. The overall confidence score is obtained by calculating the visibility score, image sharpness score and temporal stability score of all views and then using a weighted summation method. The hand motion redirection module selects the viewpoint image with the highest confidence score as the main viewpoint, extracts the coordinates of the key points of the hand, and calculates the target angles of each joint of the robotic hand by constructing an optimization objective function to minimize the difference between the operator's hand and the robotic fingertip vectors. The execution control module is used to synchronously send the target angles of each joint of the robotic arm and the joint motion angles of the robotic arm to the simulation or physical robot platform, driving it to complete the operation task.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the arm-hand robot teleoperation method based on multi-view confidence fusion as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the teleoperation method for an arm-handed robot based on multi-view confidence fusion as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the arm-hand robot teleoperation method based on multi-view confidence fusion according to any one of claims 1-6.

Images