A control method and system for a dual-arm robot based on 3D machine vision

Abstract

This invention discloses a dual-arm robot control method and system based on 3D machine vision. The method includes: point cloud segmentation acquisition: projecting Gray code onto an initial object and acquiring an initial projection image to obtain the initial 3D data and target point cloud of the object; target pose acquisition: acquiring the input point cloud of the object to be operated by the dual-arm robot and the operation position, and performing point cloud matching in combination with the target point cloud to obtain the target pose; grasping control: controlling one of the robotic arms of the dual-arm robot to grasp according to the target pose; tracking and imaging: continuously projecting Gray code onto the object grasped by the dual-arm robot and acquiring several object grasping projection images; updating target pose acquisition: performing 3D reconstruction on the acquired object grasping projection images, calculating the pose after the operation position offset, and obtaining the updated target pose; operation control: controlling the other robotic arm to operate according to the updated target pose.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a dual-arm robot control method and system based on three-dimensional machine vision. Background Technology

[0002] Most existing dual-arm robots are controlled based on static rigid bodies. When dealing with flexible objects, the objects deform during operation, causing the target points to change and making it impossible to perform the task properly. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a dual-arm robot control method and system based on three-dimensional machine vision, so as to continuously detect and track objects and provide accurate dual-arm robot control.

[0004] In a first aspect, the present invention provides a dual-arm robot control method based on three-dimensional machine vision, comprising the following steps: point cloud segmentation acquisition: projecting Gray code onto an initial object and acquiring an initial projection image to obtain the initial three-dimensional data of the object and the target point cloud; target pose acquisition: acquiring the input point cloud of the object to be operated by the dual-arm robot and the operation position, and performing point cloud matching in combination with the target point cloud to obtain the target pose; grasping control: controlling one of the robotic arms of the dual-arm robot to grasp according to the target pose; tracking and imaging: continuously projecting Gray code onto the object grasped by the dual-arm robot and acquiring several object grasping projection images; updating target pose acquisition: performing three-dimensional reconstruction on the acquired object grasping projection images, calculating the pose after the operation position offset, and obtaining the updated target pose; operation control: controlling the other robotic arm to operate according to the obtained updated target pose.

[0005] Secondly, the present invention also provides a dual-arm robot control system based on three-dimensional machine vision, including a segmentation point cloud acquisition module, a target pose acquisition module, a grasping control module, a tracking and imaging module, an updated target pose acquisition module, and an operation control module. The segmentation point cloud acquisition module is used to project Gray code onto an initial object and acquire an initial projection image, thereby acquiring the initial three-dimensional data of the object and the target point cloud. The target pose acquisition module is used to acquire the input point cloud of the object to be operated by the dual-arm robot and the operation position, and to perform point cloud matching in combination with the target point cloud to obtain the target pose. The grasping control module is used to control one of the robotic arms of the dual-arm robot to grasp according to the target pose. The tracking and imaging module is used to continuously project Gray code onto the object grasped by the dual-arm robot and acquire several object grasping projection images. The updated target pose acquisition module is used to perform three-dimensional reconstruction on the acquired object grasping projection images, calculate the pose after the operation position offset, and acquire the updated target pose. The operation control module is used to control the other robotic arm to operate according to the acquired updated target pose.

[0006] The beneficial technical effects of this invention are as follows: This invention provides a dual-arm robot control method based on 3D machine vision. It obtains 3D data by projecting Gray code and acquiring projected images, and then performs 3D reconstruction to obtain a 3D point cloud. Gray code technology improves the speed and accuracy of 3D image reconstruction. By continuously tracking and capturing images of the grasped object, it obtains projected images of the grasped object and performs 3D reconstruction to calculate the pose after the operation position shift and obtain the updated target pose. This enables continuous tracking of the target point cloud, solving the problem of point position shift caused by object deformation, and improving the accuracy and reliability of dual-arm robot operation control. The dual-arm robot control system based on 3D machine vision of this invention also has the above-mentioned functions. Attached Figure Description

[0007] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 A flowchart illustrating the dual-arm robot control method based on three-dimensional machine vision provided in an embodiment of the present invention;

[0009] Figure 2 A schematic diagram of a sub-process of a dual-arm robot control method based on three-dimensional machine vision provided in an embodiment of the present invention;

[0010] Figure 3This is a schematic diagram of the framework of a dual-arm robot control system based on three-dimensional machine vision provided in an embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] Please see Figure 1 , Figure 1 This is a flowchart illustrating a dual-arm robot control method based on 3D machine vision provided in an embodiment of the present invention. The dual-arm robot control method based on 3D machine vision includes steps S11-S16:

[0013] Step S11, Segmentation point cloud acquisition: Project Gray code onto the initial object and acquire the initial projected image to obtain the initial 3D data of the object and the target point cloud;

[0014] Step S12, Target Pose Acquisition: Acquire the input point cloud of the object to be operated by the dual-arm robot and the operation position, and perform point cloud matching in combination with the target point cloud to obtain the target pose;

[0015] Step S13, Grasping Control: Control one of the robotic arms of the dual-arm robot to grasp the target according to its pose;

[0016] Step S14, Tracking and Filming: Continuously project Gray code onto the object grasped by the dual-arm robot and acquire several object grasping projection images;

[0017] Step S15: Update target pose acquisition: Perform 3D reconstruction on the acquired object grasping projection image, calculate the pose after the operation position offset, and obtain the updated target pose; wherein, the method of dynamically generating point cloud can be used to perform 3D reconstruction on the acquired object grasping projection image, so as to more accurately track the position and posture of the object, thereby improving the operation efficiency of the dual-arm robot.

[0018] Step S16, Operation Control: Control another robotic arm to perform operations based on the obtained updated target pose.

[0019] The dual-arm robot control method based on 3D machine vision obtains 3D data and 3D point cloud by projecting Gray code and acquiring projected images, and then reconstructs 3D data. Gray code technology is used to improve the speed and accuracy of 3D image reconstruction. The method continuously tracks and captures the grasped object to obtain the object's grasped projected image and performs 3D reconstruction to calculate the pose after the operation position shift and obtain the updated target pose. By continuously acquiring the target's point cloud data, the method dynamically generates point clouds, realizes continuous tracking of the target point cloud, more accurately tracks the object's position and posture, solves the problem of point position shift caused by object deformation, and improves the accuracy and reliability of dual-arm robot operation control.

[0020] Specifically, the step S11 is preceded by:

[0021] The calibration plate is fixed to the end effectors of the two robotic arms of the dual-arm robot. The camera parameters of the 3D camera are adjusted to adapt to the current ambient light. The two robotic arms are then controlled to complete stereo calibration and hand-eye calibration respectively. Hand-eye calibration refers to the process of measuring and determining the relative pose relationship between the actuators at the end effectors of the robotic arms and the 3D camera. Hand-eye calibration requires determining the transformation matrix between the base of the robotic arm and the base of the camera, as well as the transformation matrix between the actuators at the end effectors of the robotic arms and the camera. This can be obtained by taking pictures of the calibration plate at the end effectors of the robotic arms in different poses.

[0022] Combination Figure 2 Specifically, step S11 includes:

[0023] Step S111: Project Gray code onto the initial object and acquire the initial projected image;

[0024] Step S112: Decode the acquired initial projection image to obtain the initial three-dimensional data of the object, and reconstruct the initial three-dimensional point cloud of the object.

[0025] Step S113: Segment the obtained initial 3D point cloud of the object to obtain the target point cloud. The target point cloud refers to the point cloud obtained after segmentation.

[0026] Specifically, step S113 includes:

[0027] The obtained initial 3D point cloud of the object is used to construct a KD tree using a nearest neighbor query algorithm based on the KD tree;

[0028] Traverse the KD tree to find the nearest neighbor points of the 3D point cloud of the initial object;

[0029] An adaptive K-value selection method is employed to determine the optimal K-value based on the relationship between the sum of squares within a cluster and the number of clusters. The initial 3D point cloud of the object is then segmented using this optimal K-value to obtain the target point cloud. Here, K refers to the number of target clusters to be segmented, and the sum of squares within a cluster (SSE) is the sum of the squared distances of all data points within a cluster to the centroid of that cluster. Images typically display an "elbow" point, after which the rate of decrease in the SSE within a cluster slows dramatically. The elbow point is generally considered the optimal K-value. Finding the nearest neighbor reduces computation and speeds up subsequent segmentation. The adaptive K-value selection method, by obtaining the optimal K-value, enables more accurate point cloud classification and achieves better object detection results. Furthermore, the adaptive K-value selection method can adaptively select the neighborhood size based on the local features of the point cloud, improving segmentation accuracy and efficiency.

[0030] Specifically, step S12 includes:

[0031] Target point cloud acquisition: Acquire the point cloud of the object to be operated by the dual-arm robot and the operation position;

[0032] Point cloud matching pose calculation: Perform point cloud matching pose calculation on the target point cloud based on the point cloud of the object to be operated on, and obtain the target pose.

[0033] Specifically, the point cloud matching pose calculation steps include:

[0034] Two-dimensional contour recognition is performed on the acquired initial projection image to obtain the initial object contour. The obtained initial object contour is mapped to the target point cloud to obtain the contour point cloud and its tangent vector. The initial object three-dimensional point cloud is downsampled to the size of the contour point cloud to reduce the amount of point cloud computing and improve the matching speed. Among these methods, the use of two-dimensional contour recognition can greatly improve the matching speed and make the recognition and matching results more accurate.

[0035] The PPF algorithm uses tangent vectors instead of normal vectors to match the contour point cloud and the target point cloud, obtaining a coarsely matched object pose. The features extracted through tangent vectors better reflect the local geometric information of the point cloud and are more stable and reliable. The contour point cloud is only used to obtain the coarsely matched object pose.

[0036] The target pose is obtained by calculating based on the coarsely matched object pose using the ICP algorithm.

[0037] The ICP algorithm employs a pyramid structure for multi-scale matching. It downsamples the original point cloud data level by level to generate each level of the pyramid, with corresponding point cloud data for each level. Preferably, the point cloud data at each level is downsampled to 1 / 2 or 1 / 4 of the original point cloud data, making the matching process more efficient. This pyramid-structure-based, level-by-level matching optimizes the point cloud registration process, resulting in better stability and enabling matching at different scales. This allows for better adaptation to target objects of different sizes and shapes, improving both accuracy and efficiency.

[0038] The specific steps for calculating and obtaining the target pose using the ICP algorithm based on the obtained coarsely matched object pose are as follows:

[0039] Initialization: It is assumed that there is no initial correspondence between the template point cloud and the target point cloud; where the template point cloud refers to the point cloud of the object to be operated on, which can be generated and stored according to the 3D model of the object to be operated on for later retrieval, and the operation position is the specific operation point on the template point cloud.

[0040] Corresponding point pair search: For each point in the template point cloud, search for the nearest point in the target point cloud using a certain distance metric (such as Euclidean distance), establish the starting point correspondence, and obtain the initial corresponding point pair.

[0041] Point cloud registration and correspondence update: Using the initial corresponding point pairs, estimate the rigid body transformation, transform the points in the template point cloud to the target point cloud coordinate system, obtain the transformed point cloud, and re-search for the point in the target point cloud that is closest to the transformed point cloud, updating the correspondence between the point clouds; wherein, by estimating the rigid body transformation, the overlap between the template point cloud and the target point cloud can be maximized. Rigid body transformation includes translation and rotation.

[0042] Iterative optimization: Repeat the steps of point cloud registration and updating the correspondence until a stopping criterion is met. The stopping criterion can be that the number of iterations reaches a set threshold or the change in rigid body transformation is less than a set threshold.

[0043] Target pose calculation: Obtain the final rigid body transformation at the point where iteration stops, and combine it with the coarse-matched object pose to calculate the target pose. The final rigid body transformation at the point where iteration stops describes the optimal matching relationship between the template point cloud and the target point cloud.

[0044] Please see Figure 3 , Figure 3This is a schematic diagram of the framework of a dual-arm robot control system based on 3D machine vision provided in an embodiment of the present invention. The dual-arm robot control system 10 based on 3D machine vision includes a segmentation point cloud acquisition module 11, a target pose acquisition module 12, a grasping control module 13, a tracking and shooting module 14, a target pose update acquisition module 15, and an operation control module 16. The segmentation point cloud acquisition module 11 is used to project Gray code onto an initial object and acquire an initial projected image, thereby obtaining the initial 3D data of the object and the target point cloud. The target pose acquisition module 12 is used to acquire the input object to be operated by the dual-arm robot. The system combines point cloud data and operation position data, and performs point cloud matching with the target point cloud to obtain the target pose; the grasping control module 13 controls one of the robotic arms of the dual-arm robot to grasp the object according to the target pose; the tracking and imaging module 14 continuously projects Gray code onto the object grasped by the dual-arm robot and acquires several object grasping projection images; the updated target pose acquisition module 15 performs 3D reconstruction on the acquired object grasping projection images, calculates the pose after the operation position offset, and obtains the updated target pose; the operation control module 16 controls the other robotic arm to operate according to the obtained updated target pose.

[0045] The dual-arm robot's two robotic arms can both be Elite EC66 six-DOF collaborative robots. The Elite EC66 weighs 17.5 kg, has a payload capacity of up to 6 kg, a working radius of 914 mm, and a repeatability of ±0.02 mm. Each robotic arm can be equipped with a gripper, which can be replaced according to different needs. The dual-arm robot control system 10, based on 3D machine vision, obtains 3D data and performs 3D reconstruction by projecting Gray code and acquiring projected images to obtain a 3D point cloud. Gray code technology improves the speed and accuracy of 3D image reconstruction. The tracking and capturing module 14, in conjunction with the target pose acquisition module 15, continuously tracks and captures the grasped object to obtain the object's grasped projected image and performs 3D reconstruction to calculate the pose after the operation position shift and obtain the updated target pose. This enables continuous tracking of the target point cloud, solving the problem of point position shift caused by object deformation and improving the accuracy and reliability of the dual-arm robot's operation control.

[0046] In summary, the dual-arm robot control method based on 3D machine vision of this invention obtains 3D data and 3D point cloud by projecting Gray code and acquiring projected images, and then reconstructs the 3D image. Gray code technology improves the speed and accuracy of 3D image reconstruction. By continuously tracking and capturing images of the grasped object to obtain object-grabbed projected images and performing 3D reconstruction, the pose after the operation position shift is calculated and the updated target pose is obtained. This enables continuous tracking of the target point cloud, solving the problem of point position shift caused by object deformation, and improving the accuracy and reliability of dual-arm robot operation control. The dual-arm robot control system based on 3D machine vision of this invention also has the above-mentioned functions.

[0047] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A control method for a dual-arm robot based on three-dimensional machine vision, characterized in that, Includes the following steps: Point cloud segmentation: Project Gray code onto the initial object and acquire the initial projected image to obtain the initial 3D data of the object and the target point cloud; Target pose acquisition: Acquire the point cloud of the object to be operated by the dual-arm robot and the operation position, and perform point cloud matching in combination with the target point cloud to obtain the target pose; Grasping control: Control one of the robotic arms of the dual-arm robot to grasp the target according to its pose; Tracking and filming: Continuously project Gray codes onto the objects grasped by the dual-arm robot and collect several object grasping projection images; Update target pose acquisition: Perform 3D reconstruction on the captured object capture projection image, calculate the pose after the operation position offset, and obtain the updated target pose; Operation control: Control another robotic arm to perform operations based on the obtained updated target pose; The steps for obtaining the segmented point cloud specifically include: Gray code is projected onto the initial object and an initial projected image is acquired; The acquired initial projection image is decoded to obtain the initial three-dimensional data of the object, and the initial three-dimensional point cloud of the object is obtained by three-dimensional reconstruction. The obtained initial three-dimensional point cloud of the object is segmented to obtain the target point cloud; The step of segmenting the obtained initial three-dimensional point cloud of the object to obtain the target point cloud specifically includes: The obtained initial 3D point cloud of the object is used to construct a KD tree using a nearest neighbor query algorithm based on the KD tree; Traverse the KD tree to find the nearest neighbor points of the 3D point cloud of the initial object; An adaptive K-value selection method is adopted to determine the optimal K-value based on the relationship between the sum of squares within a cluster and the number of clusters. The initial object's 3D point cloud is then segmented based on the optimal K-value to obtain the target point cloud.

2. The dual-arm robot control method based on three-dimensional machine vision according to claim 1, characterized in that, The step of obtaining the segmented point cloud also includes: Fix the calibration plate to the ends of the two robotic arms of the dual-arm robot, adjust the camera parameters of the 3D camera, and control the two robotic arms to complete the stereo calibration and hand-eye calibration respectively.

3. The dual-arm robot control method based on three-dimensional machine vision according to claim 1, characterized in that, The steps for obtaining the target pose include: Target point cloud acquisition: Acquire the point cloud of the object to be operated by the dual-arm robot and the operation position; Point cloud matching pose calculation: Perform point cloud matching pose calculation on the target point cloud based on the point cloud of the object to be operated on, and obtain the target pose.

4. The dual-arm robot control method based on three-dimensional machine vision according to claim 3, characterized in that, The steps for point cloud matching pose calculation include: Two-dimensional contour recognition is performed on the acquired initial projection image to obtain the initial object contour. The obtained initial object contour is mapped to the target point cloud to obtain the contour point cloud and the tangent vector of the contour point cloud. The initial object three-dimensional point cloud is downsampled to the size of the contour point cloud. According to the PPF algorithm, the tangent vector is used instead of the normal vector to match the contour point cloud and the target point cloud to obtain the coarse matching object pose; The target pose is obtained by calculating based on the coarsely matched object pose using the ICP algorithm.

5. The dual-arm robot control method based on three-dimensional machine vision according to claim 4, characterized in that, In the step of calculating the target pose using the ICP algorithm based on the obtained coarsely matched object pose, the ICP algorithm adopts a pyramid structure.

6. A dual-arm robot control system based on three-dimensional machine vision, characterized in that, include: The segmentation point cloud acquisition module is used to project Gray code onto the initial object and acquire the initial projected image, thereby obtaining the initial 3D data of the object and the target point cloud. The target pose acquisition module is used to acquire the point cloud of the object to be operated by the dual-arm robot and the operation position, and to perform point cloud matching in combination with the target point cloud to obtain the target pose. The grasping control module is used to control one of the robotic arms of the dual-arm robot to grasp the target according to its pose. The tracking and shooting module is used to continuously project Gray codes onto the objects grasped by the dual-arm robot and acquire several object grasping projection images. The target pose acquisition module is updated to perform three-dimensional reconstruction on the captured object capture projection image, calculate the pose after the operation position offset, and obtain the updated target pose. The operation control module is used to control another robotic arm to perform operations based on the obtained updated target pose; Specifically, the segmentation point cloud acquisition module is used to project Gray code onto an initial object and acquire an initial projected image; decode the acquired initial projected image to obtain the initial three-dimensional data of the object, and reconstruct the initial three-dimensional point cloud of the object; segment the acquired initial three-dimensional point cloud of the object to obtain a target point cloud; the segmentation of the acquired initial three-dimensional point cloud of the object to obtain the target point cloud specifically includes: constructing a KD tree based on a KD tree-based nearest neighbor query algorithm for the acquired initial three-dimensional point cloud of the object; traversing the KD tree to find the nearest neighbor points of the initial three-dimensional point cloud of the object; using an adaptive K-value selection method, determining the optimal K-value based on the relationship graph between the sum of squares within a cluster and the number of clusters, and segmenting the initial three-dimensional point cloud of the object according to the optimal K-value to obtain the target point cloud.

Images