Arm-hand robot teleoperation method and system based on multi-view confidence fusion

Abstract

The invention provides an arm-hand robot teleoperation method and system based on multi-view confidence fusion, and relates to the technical field of robots. According to the method, related information displayed at a plurality of different visual angles is fully scored and discriminated, the confidence degree of the different visual angles is determined, so that better selection can be carried out in a plurality of camera visual angles, the problem caused by hand self-shielding is relieved, the problem of hand self-shielding in smart operation is solved, and the user experience is improved. Therefore, the robustness, accuracy and efficiency of remote control operation based on vision are improved while low cost and high adaptability are kept, and powerful support is provided for smart operation research based on imitation learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, in particular to a multi-view confidence fusion-based robotic arm teleoperation method and system. BACKGROUND

[0002] Today, the development of robot technology is rapid, and teleoperation shows a compelling effect on the dexterous operation of robotic arms, which can overcome the control difficulty caused by high degrees of freedom, so that the operator can solve various complex operation tasks through this way. At the same time, it provides an effective and intuitive way to collect robot demonstration data. Especially after the learning method from demonstration has made great progress in recent years, the teleoperation method has attracted much attention. Traditional teleoperation is mainly used in environments that humans cannot reach, such as space exploration, disaster rescue, and remote medical operation. The recent research focus has shifted to collecting demonstration data through teleoperation for policy learning to achieve autonomous task completion. For robotic arms with dexterous hands, due to the excessive degrees of freedom and the difference between the mechanical arm and the human body configuration, the existing methods use VR devices, exoskeleton devices, motion capture devices, etc. The above-mentioned methods usually face challenges such as being unfriendly to operators and high material costs, but for traditional visual teleoperation systems, this method is easily affected by hand self-occlusion and environmental factors, which reduces the operation accuracy. It is difficult to meet the requirements of convenience, speed and economy.

[0003] Therefore, it is necessary to provide a multi-view confidence fusion-based robotic arm teleoperation method and system to solve the above technical problems. SUMMARY

[0004] To solve the above technical problems, the present application provides a multi-view confidence fusion-based robotic arm teleoperation method and system to solve the problem of existing technology that is easily affected by hand self-occlusion and environmental factors, which reduces the operation accuracy.

[0005] The multi-view confidence fusion-based robotic arm teleoperation method provided by the present application comprises the following steps: S1, synchronously collecting hand images of an operator through multiple cameras at different viewing angles to form multi-view video stream data; S2, obtaining wrist key points and finger root key points from the multi-view video stream data, constructing a 6D pose after mixing filtering, and solving the joint motion angle of the robotic arm through an inverse kinematics solver; S3, calculating the confidence score of all viewing angles in all multi-view video stream data, wherein the confidence score is obtained by calculating the visibility score, image clarity score and time stability score of all viewing angles, and calculating by using weighted summation; S4, select the view image with the highest comprehensive confidence score as the main view, extract the hand key point coordinates, and solve the minimum difference between the operator's hand and the mechanical finger tip vector by constructing an optimization objective function to calculate the target angle of each joint of the mechanical hand; S5, synchronously send the target angle of each joint of the mechanical hand and the joint motion angle of the mechanical arm to the simulation or physical robot platform to drive it to complete the operation task.

[0006] Preferably, step S1 specifically comprises: synchronously collecting hand images of the operator from different angles by configuring at least one RGB-D camera and at least one RGB camera, and aligning the hand images from multiple angles according to the time axis to form a multi-view video stream data.

[0007] Preferably, the specific steps of step S2 are: S201, the wrist key points and finger root key points data collected by the RGB-D camera in the multi-view video stream data are mixed filtered, wherein the mixed filtering specifically includes spatial median filtering, outlier rejection and previous value filling, Kalman filtering and moving average filtering; S202, map the wrist key points after mixed filtering to the target position of the end of the mechanical arm; S203, calculate the local coordinate axis according to the wrist key points after mixed filtering and the index finger root key point and the little finger root key point in the finger root key point data, and map it to the target pose of the end of the mechanical arm; S204, combine 6D pose according to the target position and target pose of the end of the mechanical arm, and solve the joint motion angle of the mechanical arm according to the inverse kinematics solver.

[0008] Preferably, in step S3, the visibility score includes: obtaining the current visible hand key points from the multi-view video stream data, and calculating the ratio of the convex hull area formed based on the current visible hand key points to the theoretical complete hand area, which is used to evaluate the degree of occlusion; The image sharpness score includes: calculating the ratio of the image Laplacian variance to the preset maximum variance, which is used to evaluate the image blur degree; The time stability score includes: calculating the normalized value based on the average displacement of the hand key points of the current frame and the previous frame, wherein the smaller the displacement, the higher the stability.

[0009] Preferably, in step S4, the calculation formula of the optimization objective function specifically includes: ; In the formula, represents the joint position of the dexterous hand at time t in the motion control process, represents the i-th key point vector of the hand calculated according to the detected finger key points, is the ith forward kinematics function, taking the manipulator joint positions as output, is a scaling factor to account for hand size differences; is a penalty term to increase temporal smoothness.

[0010] The view angle image with the highest confidence score is selected as the main view angle, and the hand key point coordinates thereof are extracted, specifically including extracting 2.5D coordinates of 21 hand key points by using a MediaPipe algorithm.

[0011] The application further provides an arm-hand robot teleoperation system based on multi-view angle confidence fusion, comprising: A data acquisition module is configured to synchronously acquire hand images of an operator by multiple cameras at different view angles to form multi-view angle video stream data. An inverse kinematics solving module is configured to acquire wrist key points and finger root key points from the multi-view angle video stream data, construct 6D poses after mixed filtering, and obtain joint motion angles of a manipulator by an inverse kinematics solver. A confidence evaluation module is configured to calculate confidence comprehensive scores of all view angles in all multi-view angle video stream data, wherein the confidence comprehensive scores are obtained by calculating visibility scores, image definition scores and temporal stability scores of all view angles and by using a weighted summation method. A hand action redirection module is configured to select a view angle image with the highest confidence comprehensive score as a main view angle, extract hand key point coordinates thereof, and calculate target angles of joints of the manipulator by constructing an optimization objective function to minimize vector differences between the hand of the operator and the fingers of the manipulator. An execution control module is configured to synchronously send the target angles of the joints of the manipulator and the joint motion angles of the manipulator to a simulation or physical robot platform to drive the simulation or physical robot platform to complete an operation task.

[0012] The application further provides an electronic device, comprising at least one processor and a memory connected with the at least one processor in communication, wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the arm-hand robot teleoperation method based on multi-view angle confidence fusion.

[0013] The application further provides a computer readable storage medium storing computer instructions for enabling a processor to execute the arm-hand robot teleoperation method based on multi-view angle confidence fusion when the computer instructions are executed by the processor.

[0014] The present invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the teleoperation method for an arm-handed robot based on multi-view confidence fusion.

[0015] Compared with related technologies, the teleoperation method and system for arm-handed robots based on multi-view confidence fusion provided by this invention have the following beneficial effects: This invention fully scores and judges relevant information displayed from multiple different perspectives, determines the confidence level of different perspectives, and thus makes better selection among multiple camera perspectives. This alleviates the problem caused by hand self-occlusion and overcomes the problem of hand self-occlusion in dexterity operation. In this way, while maintaining low cost and high adaptability, it improves the robustness, accuracy and efficiency of vision-based teleoperation, and provides strong support for the research of dexterity operation based on imitation learning. Attached Figure Description

[0016] Figure 1 This is a flowchart of the arm-hand robot teleoperation method based on multi-view confidence fusion according to the present invention. Figure 2 This is a system block diagram of the arm-handed robot teleoperation system based on multi-view confidence fusion according to the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0018] Although the steps in this invention are arranged by reference numerals, this is not intended to limit the order of the steps. Unless the order of the steps is explicitly stated or the execution of a step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items.

[0019] Example 1 like Figure 1 As shown, the teleoperation method for a hand-arm robot based on multi-view confidence fusion includes the following steps: S1. Simultaneously capture images of the operator's hands from different perspectives using multiple cameras to form multi-view video stream data.

[0020] Specifically, by configuring at least one RGB-D camera and at least one RGB camera, images of the operator's hands are simultaneously acquired from different perspectives, and the hand images from multiple perspectives are aligned along the timeline and combined to form multi-view video stream data.

[0021] S2. Obtain wrist key points and finger root key points from multi-view video stream data, construct 6D pose after mixing and filtering, and solve the joint motion angles of the robotic arm through an inverse kinematics solver.

[0022] In the specific implementation process, the specific steps of step S2 are as follows: S201. The wrist key point and finger root key point data acquired by the RGB-D camera in the multi-view video stream data are mixed and filtered. The mixed filtering specifically includes spatial median filtering, outlier removal and pre-value filling, Kalman filtering and moving average filtering.

[0023] S202. Map the wrist key points after hybrid filtering to the target position at the end of the robotic arm.

[0024] S203. Based on the wrist key points and the key points at the base of the index finger and the base of the little finger in the mixed-filtered wrist key points and finger root key points data, calculate the local coordinate axes and map them to the target posture at the end of the robotic arm.

[0025] S204. Based on the target position and target pose combination of the robotic arm end effector, a 6D pose is obtained, and the joint motion angles of the robotic arm are obtained by solving the 6D pose inverse kinematics solver.

[0026] Specifically, the 6D pose of the wrist is obtained and mapped onto the end effector of the robotic arm to define the wrist position information. and wrist rotation information The wrist key point location information is transformed into 3D coordinates from the 2.5D wrist key points identified by Mediapipe through a hybrid filtering process. .

[0027] Use the same procedure to obtain the base of the index finger. and the base of the little finger The spatial coordinates are obtained by normalizing the vectors from the wrist key point to the key point at the base of the index finger and to the key point at the base of the little finger, and using them as two axes. and Construct a plane parallel to the palm. The specific calculation formula can be expressed as follows: ; in, This indicates a normalization operation.

[0028] The third axis perpendicular to the plane is obtained through the cross product. The specific calculation formula can be expressed as: ; in, This represents the cross product operation.

[0029] Rotation at key wrist points obtained through orthogonal combination , can be represented as: ; And combined with the motion vector to determine the complete 6D wrist pose. , can be represented as: ; The human wrist posture The target pose of the robotic arm's end effector is mapped onto the end effector of the robotic arm. The change of this coordinate over time is mapped to the change in the direction of the end effector. Then, the inverse kinematics of the arm is solved using mink, a Python differential inverse kinematics library based on the MuJoCo physics engine and the Pinocchio inverse kinematics library. It can generate smooth and feasible motions for the robotic arm and finally obtain the joint angles of the robotic arm.

[0030] S3. Calculate the overall confidence score for all views in all multi-view video stream data. The overall confidence score is obtained by calculating the visibility score, image sharpness score and temporal stability score of all views and then using a weighted summation method.

[0031] Specifically, the formula for calculating the visibility score includes: ; ; In the formula, This represents the area of ​​the currently visible hand, specifically the area of ​​the polygon enclosed by the currently visible hand keypoints. The number of polygon vertices involved in the area calculation is 21 in this embodiment. These are the location information of key hand points identified by Mediapipe. This is an estimate of the palm width, mainly measured from the base of the index finger and little finger. and The estimation is based on the distance between key points. This is an estimate of the hand length, calculated from the wrist position and the base of the middle finger. and The estimation is based on the distance between key points. It is an estimated complete theoretical area of ​​the hand, where and The width and length of the image are 640 and 480, respectively.

[0032] Image sharpness score, the formula for which is calculated includes: ; In the formula, It's a perspective The variance of the image Laplacian operator, Used for normalization, the values ​​are determined based on the parameters of cameras with the same viewpoint. In this embodiment, the values ​​for the two viewpoints used are 200 and 400, respectively. The clearer the image, the better. The closer the score is to 1, the more reliable the pose. This is used to evaluate the impact of image sharpness on image credibility from different viewpoints.

[0033] The time stability score is calculated using the following formula: ; In the formula, This sets the key points on the wrist in the current frame and the previous frame. It is the average displacement where It is a scaling parameter that controls the influence of displacement confidence. Small displacements (close to 1) indicate stable results.

[0034] The weighted summation method is specifically calculated using the following formula: ; In the formula, , , All are weighting coefficients and In this embodiment, the weighting coefficient is set. , , .

[0035] S4. Select the viewpoint image with the highest confidence score as the main viewpoint, extract the coordinates of the key points of the hand, and calculate the target angle of each joint of the robot hand by constructing an optimization objective function to minimize the difference between the operator's hand and the mechanical fingertip vector.

[0036] Specifically, based on the coordinates of 21 key hand points in the corresponding image, the vector between the fingertips of the operator's hand is calculated, and its norm is calculated with a vector between the fingertips of the robotic hand. The optimal solution is then obtained through optimization to determine the angle of the robotic hand's motor joints. The specific calculation formula is as follows: ; In the formula, This indicates the joint position during time t in the movement control process of a dexterous hand. This represents the vector of the i-th key point of the hand calculated based on the detected finger key points. It is the i-th positive kinematics function, with the joint positions of the robot hand as the output. It is a scaling factor used to account for differences in hand size; It is a penalty item used to improve time smoothness; The viewpoint image with the highest overall confidence score was selected as the main viewpoint, and the coordinates of its hand key points were extracted. Specifically, the 2.5D coordinates of 21 hand key points were extracted using the MediaPipe algorithm. The MediaPipe algorithm is a lightweight hand detection tool based on RGB that processes multi-view images and can run in real time on the CPU. It is used to detect 21 hand key points in the wrist frame and obtain the 2.5D coordinates of all 21 key points. The hand key point numbers are represented by 0, 1, k, and 20.

[0037] S5. The target angles of each joint of the robotic arm and the joint motion angles of the robotic arm are synchronously sent to the simulation or physical robot platform to drive it to complete the operation task.

[0038] Example 2 like Figure 2 As shown, the arm-handed robot teleoperation system based on multi-view confidence fusion includes: The data acquisition module is used to simultaneously acquire images of the operator's hands from different perspectives using multiple cameras, forming multi-view video stream data; The inverse kinematics solving module is used to obtain wrist key points and finger root key points from multi-view video stream data, construct 6D pose after mixing and filtering, and solve the joint motion angles of the robotic arm through the inverse kinematics solver. The confidence assessment module is used to calculate the overall confidence score of all views in all multi-view video stream data. The overall confidence score is obtained by calculating the visibility score, image sharpness score and temporal stability score of all views and then using a weighted summation method. The hand motion redirection module selects the viewpoint image with the highest confidence score as the main viewpoint, extracts the coordinates of the key points of the hand, and calculates the target angles of each joint of the robotic hand by constructing an optimization objective function to minimize the difference between the operator's hand and the robotic fingertip vectors. The execution control module is used to synchronously send the target angles of each joint of the robotic arm and the joint motion angles of the robotic arm to the simulation or physical robot platform, driving it to complete the operation task.

[0039] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0040] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0041] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

Claims

1. A teleoperation method for a hand-arm robot based on multi-view confidence fusion, characterized in that, Includes the following steps: S1. Simultaneously capture images of the operator's hands from different perspectives using multiple cameras to form multi-view video stream data; S2. Obtain wrist key points and finger root key points from multi-view video stream data, construct 6D pose after mixing and filtering, and solve the joint motion angles of the robotic arm through an inverse kinematics solver. S3. Calculate the overall confidence score for all views in all multi-view video stream data. The overall confidence score is obtained by calculating the visibility score, image sharpness score and temporal stability score of all views and then using a weighted summation method. S4. Select the viewpoint image with the highest confidence score as the main viewpoint, extract the coordinates of the key points of the hand, and calculate the target angle of each joint of the robot hand by constructing an optimization objective function to minimize the difference between the operator's hand and the mechanical fingertip vector. S5. The target angles of each joint of the robotic arm and the joint motion angles of the robotic arm are synchronously sent to the simulation or physical robot platform to drive it to complete the operation task.

2. The teleoperation method for a hand-arm robot based on multi-view confidence fusion according to claim 1, characterized in that, Step S1 specifically includes: by configuring at least one RGB-D camera and at least one RGB camera, synchronously acquiring images of the operator's hand from different perspectives, aligning the hand images from multiple perspectives according to the timeline, and combining them to form multi-view video stream data.

3. The teleoperation method for a hand-arm robot based on multi-view confidence fusion according to claim 1, characterized in that, The specific steps of step S2 are as follows: S201. The wrist key point and finger root key point data acquired by the RGB-D camera in the multi-view video stream data are mixed and filtered. The mixed filtering specifically includes spatial median filtering, outlier removal and pre-value filling, Kalman filtering and moving average filtering. S202. Map the wrist key points after hybrid filtering to the target position at the end of the robotic arm; S203. Based on the wrist key points and the key points at the base of the index finger and the base of the little finger in the mixed-filtered wrist key points and finger root key points data, calculate the local coordinate axes and map them to the target posture at the end of the robotic arm. S204. Based on the target position and target pose combination of the robotic arm end effector, a 6D pose is obtained, and the joint motion angles of the robotic arm are obtained by solving the 6D pose inverse kinematics solver.

4. The teleoperation method for a hand-arm robot based on multi-view confidence fusion according to claim 1, characterized in that, In step S3, the visibility score includes: obtaining the currently visible hand key points from the multi-view video stream data, and calculating the ratio of the convex hull area formed by the currently visible hand key points to the theoretical complete hand area, which is used to evaluate the degree of occlusion; the image sharpness score includes: obtaining the ratio of the image Laplacian variance to the preset maximum variance, which is used to evaluate the degree of image blur; the temporal stability score includes: calculating the normalized value of the average displacement of the hand key points between the current frame and the previous frame, where a smaller displacement indicates higher stability.

5. The teleoperation method for a hand-arm robot based on multi-view confidence fusion according to claim 1, characterized in that, In step S4, the specific formula for calculating the objective function includes: ； In the formula, This indicates the joint position during time t in the movement control process of a dexterous hand. This represents the vector of the i-th key point of the hand calculated based on the detected finger key points. It is the i-th positive kinematics function, with the joint positions of the robot hand as the output. It is a scaling factor used to account for differences in hand size; It is a penalty item used to improve time smoothness.

6. The teleoperation method for a robotic arm based on multi-view confidence fusion according to claim 1, characterized in that, In step S4, selecting the viewpoint image with the highest confidence score as the main viewpoint and extracting the coordinates of its hand key points specifically includes using the MediaPipe algorithm to extract the 2.5D coordinates of 21 hand key points.

7. A teleoperation system for a hand-arm robot based on multi-view confidence fusion, employing the teleoperation method for a hand-arm robot based on multi-view confidence fusion as described in any one of claims 1-6, characterized in that, include: The data acquisition module is used to simultaneously acquire images of the operator's hands from different perspectives using multiple cameras, forming multi-view video stream data; The inverse kinematics solving module is used to obtain wrist key points and finger root key points from multi-view video stream data, construct 6D pose after mixing and filtering, and solve the joint motion angles of the robotic arm through the inverse kinematics solver. The confidence assessment module is used to calculate the overall confidence score of all views in all multi-view video stream data. The overall confidence score is obtained by calculating the visibility score, image sharpness score and temporal stability score of all views and then using a weighted summation method. The hand motion redirection module selects the viewpoint image with the highest confidence score as the main viewpoint, extracts the coordinates of the key points of the hand, and calculates the target angles of each joint of the robotic hand by constructing an optimization objective function to minimize the difference between the operator's hand and the robotic fingertip vectors. The execution control module is used to synchronously send the target angles of each joint of the robotic arm and the joint motion angles of the robotic arm to the simulation or physical robot platform, driving it to complete the operation task.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the arm-hand robot teleoperation method based on multi-view confidence fusion as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the teleoperation method for an arm-handed robot based on multi-view confidence fusion as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the arm-hand robot teleoperation method based on multi-view confidence fusion according to any one of claims 1-6.

Images