Robot hand-eye coordinate conversion method, device, computer equipment and storage medium

Abstract

The present application relates to a robot hand-eye coordinate conversion method, device, computer device, storage medium and computer program product. The method includes: acquiring images of the end effector of the robot installed with a calibration board through a scanner of the robot to obtain calibration board images of the end effector in different poses; detecting each fiducial point in the calibration board images in different poses according to at least two patterns to obtain at least two groups of different fiducial point regions; determining whether the regional centers of the fiducial point regions in each group are valid based on the distances between the fiducial point regions in each group; generating a fiducial point sequence in different poses according to each valid regional center and the central fiducial point in different poses of each valid regional center; and calibrating the coordinate conversion relationship between the robot and the scanner through the fiducial point sequences in different poses. Using this method can improve the coordinate conversion accuracy between the scanner and the robot.

Description

Technical Field

[0001] The present application relates to the technical field of robotics, and in particular, to a method, device, computer device, storage medium, and computer program product for robot hand-eye coordinate transformation. Background Art

[0002] With the development of artificial intelligence technology, robots have been widely used in multiple industries. Among them, in the field of industrial applications, robots have a visual perception system. Using the three-dimensional information obtained by the visual perception system, the robot can control the end effector to perform actions such as machining and installation. Simply put, the three-dimensional perception system is equivalent to a person's eyes, and the end effector is equivalent to a person's hand. Through the cooperation between the hand and the eye, the preset action tasks can be completed.

[0003] In order to ensure that the robot accurately moves the spatial object to the target position, it is necessary to determine the conversion relationship between the coordinate system of the vision system and the coordinate system of the robot arm. The accuracy of the traditional method for determining the conversion relationship is poor, and the obtained result will have a large deviation from the actual true value. Summary of the Invention

[0004] Based on this, it is necessary to provide a robot hand-eye coordinate transformation method, device, computer device, computer-readable storage medium, and computer program product that can improve the accuracy in view of the above technical problems.

[0005] In a first aspect, the present application provides a method for robot hand-eye coordinate transformation. The method includes:

[0006] Performing image acquisition on the end effector of the robot installed with a calibration board through a scanner of the robot to obtain calibration board images of the end effector in different poses;

[0007] Detecting each fiducial point in the calibration board images in different poses according to at least two patterns to obtain at least two different fiducial point regions;

[0008] Based on the distances between the fiducial point regions in each group, determining whether the region centers of the fiducial point regions in each group are valid;

[0009] Generating a fiducial point sequence in different poses according to each valid region center and each valid region center's central fiducial point in the different poses;

[0010] Calibrating the coordinate system transformation relationship between the end effector and the scanner through the fiducial point sequence in different poses.

[0011] In one embodiment, each fiducial point in the calibration plate images of different poses is detected according to at least two patterns to obtain at least two different fiducial point regions, including:

[0012] For each fiducial point in the calibration plate image, edge extraction detection is respectively performed according to at least two different patterns to obtain at least two different image contours;

[0013] Screen the image contours in the contour length intervals of each group;

[0014] Based on the image contours screened in each group, the at least two different fiducial point regions are obtained.

[0015] In one embodiment, the image contours include circular contours and polygonal contours. The obtaining of the at least two different fiducial point regions based on the image contours screened in each group includes:

[0016] Calculate the similarity of the circular contours based on the contour area and contour length of the screened circular contours to obtain the circular contour similarity;

[0017] Based on the circular contour similarity, select the circular fiducial point regions corresponding to each fiducial point from the found circular contours;

[0018] Perform fitting on the polygonal contours in each found group to obtain a polygonal fitted image;

[0019] Use the quadrilateral contour in the polygonal fitted image as the quadrilateral fiducial point region corresponding to each fiducial point;

[0020] The determination of whether the region centers of the fiducial point regions in each group are valid includes:

[0021] Based on the distances between the circular fiducial point regions and the quadrilateral fiducial point regions of each fiducial point, determine whether the region center of the circular fiducial point region is valid.

[0022] In one embodiment, the determination of whether the region centers of the fiducial point regions in each group are valid based on the distances between the fiducial point regions in each group includes:

[0023] Perform overlapping detection on the fiducial point regions in each group respectively, and remove the overlapping fiducial point regions in each group to obtain the fiducial point regions after removal in each group;

[0024] Compare the distances between the fiducial point regions after removal in each group with the fiducial point adjacent threshold distance to obtain multiple adjacent point comparison results;

[0025] Based on the comparison results of each of the adjacent points, respectively determine whether the regional centers in each group of the marked point regions after rejection are valid.

[0026] In one embodiment, the overlapping detection of the marked point regions in each group is performed, and the overlapping marked point regions in each group are removed to obtain the marked point regions after rejection in each group, including:

[0027] Perform combined comparison on the marked point regions in each group respectively;

[0028] Calculate the overlapping detection distance between the marked point regions of the combined comparison;

[0029] When the overlapping detection distance meets the contour detection threshold, calculate the regional side length and regional area of each of the marked point regions of the combined comparison respectively;

[0030] Based on the regional side length and regional area of each of the marked point regions of the combined comparison, remove the overlapping marked point regions in each group to obtain the marked point regions after rejection in each group.

[0031] In one embodiment, the generating of the marked point sequences in different poses based on each of the valid regional centers and the central marked points of each of the valid regional centers in different poses includes:

[0032] Perform averaging calculation based on the valid regional centers with the same pose to obtain the centroid positions of the regional centers in different poses;

[0033] Based on the distances between the centroid positions with the same pose and the valid regional centers, find out the central marked points of each pose from each of the regional centers;

[0034] Respectively determine the central marked points of each of the regional centers in different poses as the polar coordinate origins in each of the poses;

[0035] Based on each of the regional centers and the polar coordinate origins in each of the poses, obtain the positions of each of the regional centers in the polar coordinate systems of each of the poses;

[0036] Sort the positions of each of the regional centers in the polar coordinate systems of each of the poses according to the angles of each of the regional centers in the polar coordinate systems to obtain the marked point sequences in different poses.

[0037] In one embodiment, the coordinate system conversion relationship includes a rotational conversion relationship and a translational conversion relationship. The calibration of the coordinate system conversion relationship between the robot and the scanner through the marked point sequences in different poses includes:

[0038] Calibrate the rotation conversion relationship between the robot and the scanner through the rotation vectors corresponding to the marker point sequences during different pose conversions;

[0039] Obtain the first translation vector through the origin translation information of the marker point sequences during different pose conversions;

[0040] Calculate the second translation vector through the center of the marker ball obtained by fitting the marker point sequences of different poses and the marker point sequences of a preset pose;

[0041] Combine the first translation vector and the second translation vector to obtain the translation conversion relationship.

[0042] In a second aspect, the present application also provides a robot hand-eye coordinate conversion device. The device includes:

[0043] An image acquisition module, configured to acquire images of the end effector of the robot with a calibration board installed through a scanner of the robot, and obtain calibration board images of the end effector in different poses;

[0044] An edge detection module, configured to detect each marker point in the calibration board images of different poses according to at least two patterns, and obtain at least two groups of different marker point regions;

[0045] An effective marker point determination module, configured to determine whether the region centers of the marker point regions in each group are effective based on the distances between the marker point regions in each group;

[0046] A marker point sequence generation module, configured to generate marker point sequences of different poses according to each effective region center and the center marker points of each effective region center in different poses;

[0047] A hand-eye calibration module, configured to calibrate the coordinate conversion relationship between the robot and the scanner through the marker point sequences of different poses.

[0048] In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of robot hand-eye coordinate conversion in any of the above embodiments.

[0049] In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps of robot hand-eye coordinate conversion in any of the above embodiments.

[0050] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program which, when executed by a processor, implements the steps of robot hand-eye coordinate transformation in any of the above embodiments.

[0051] For the above robot hand-eye coordinate transformation method, device, computer device, storage medium and computer program product, each fiducial point in the calibration board images with different poses is detected according to at least two patterns. Through this image geometric feature detection method, a fiducial point area is filtered out; based on the distances between the fiducial point areas in each group, it is determined whether the fiducial points in the calibration board image are valid, and then based on the center of each valid area and the center fiducial points of each valid area in different poses, a fiducial point sequence in different poses is generated, so as to determine the positions of the fiducial points in the calibration board image through the fiducial point sequence, thereby transforming the robot hand-eye calibration calculation into the calculation of the translation and rotation relationship between point pairs in three-dimensional space. The corresponding translation and rotation relationship can be calculated based on the PnP algorithm, without the need to calculate based on the change relationship between poses, and the calculation error is easy to control, which is convenient for application developers to debug. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is an application environment diagram of the robot hand-eye coordinate transformation method in an embodiment;

[0053] Figure 2 It is a flowchart of the robot hand-eye coordinate transformation method in an embodiment;

[0054] Figure 3 It is a schematic structural diagram of a fiducial board in an embodiment;

[0055] Figure 4 It is a schematic structural diagram of a fiducial point in another embodiment;

[0056] Figure 5 It is a structural block diagram of the robot hand-eye coordinate transformation device in an embodiment;

[0057] Figure 6 It is an internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0058] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] The robot hand-eye coordinate transformation method provided by the embodiments of the present application can be applied to, for example Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed in the cloud or other network servers. The terminal 102 uses the scanner of the robot to collect images of the end effector of the robot installed with a calibration board, and obtains calibration board images of the end effector in different poses; for each fiducial point in the calibration board images in different poses, at least two graphics are detected to obtain at least two groups of different fiducial point regions; based on the distances between the fiducial point regions in each group, it is determined whether the region centers of the fiducial point regions in each group are valid; according to each of the region centers and the center fiducial points of each of the region centers in different poses, a fiducial point sequence in different poses is generated; through the fiducial point sequence in different poses, the coordinate transformation relationship between the end effector and the scanner is calibrated.

[0060] Among them, the terminal 102 can be, but is not limited to, various robots, personal computers, laptop computers, smartphones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0061] In one embodiment, as Figure 2 shown, a robot hand-eye coordinate transformation method is provided. Taking the method applied to the Figure 1 terminal 102 in it as an example for description, it includes the following steps:

[0062] Step 202, using the scanner of the robot to collect images of the end effector of the robot installed with a calibration board, and obtaining calibration board images of the end effector in different poses.

[0063] The robot includes a robot body, a scanner, and the end effector of the robot; the coordinate displacement relationship between the robot body and the scanner is fixed. The scanner includes a depth map and a grayscale camera, which are used to determine the fiducial points and their pixel coordinates; at least one axis of the robot body installs the end effector of the robot, and the manipulator controlled by the end effector is installed with a fiducial board for hand-eye calibration, and the fiducial board is pasted with fiducial points. The fiducial board is a target made of lightweight aluminum alloy, and the target is pasted with coded fiducial points. A white paper should be pasted on the target first and then the fiducial points should be pasted on it to improve the recognition accuracy of the fiducial points at a long distance.

[0064] In one embodiment, the calibration board is as Figure 3As shown, there are annular position marking points arranged in a ring at the edge of the sign board. A corresponding nearby marking point is set near a certain annular position marking point as a marking position, and a central position marking point is set at the center of the sign board. In another embodiment, there are at least 9 annular position marking points, and a corresponding nearby marking point is set near a certain annular position marking point as a marking position. The angular interval between the connecting lines of each adjacent annular position marking point and the central position marking point is at least 15 degrees, serving as candidate marking points in the calibration board image and being the marking points for image detection. The angular intervals between the connecting lines of the annular position marking points and their corresponding marking positions with the central position marking point are less than 5 degrees, and the marking positions are not the marking points for image detection. The marking point is a graphic composed of multiple features, such as Figure 4 shown.

[0065] In one embodiment, the working range of a certain robotic arm corresponding to the end effector of the robot is determined. Within this working range, the end effector carrying the sign board is controlled to move multiple times to cover the working interval range as large as possible, and the scanner is used to collect images of the multiple movements of the end effector to reduce the error in subsequent calculation of pose estimation. Among them, when calculating the coordinate transformation relationship of rotation, during each movement of the end effector, the angle between the tool coordinate system and the robot body coordinate system is kept unchanged.

[0066] In one embodiment, the scanner is used to collect images of the multiple movements of the end effector, including: during the process of calculating the coordinate transformation relationship of translation, the origin position of the tool coordinate system is kept different during each movement, and the robot tool coordinate system is rotated. After the movement ends, the left and right images of the binocular camera of the scanner are used to collect two groups of images respectively, and then two groups of calibration board image sequences of the end effector in different poses are obtained; when the two groups of calibration board image sequences and the internal and external parameters in the binocular camera are calculated according to the triangulation method, the coordinate positions of each marking point in different poses are obtained.

[0067] It should be understood that using the scanner to collect images of the multiple movements of the end effector involves two processes. One process is to calculate the coordinate transformation relationship of rotation, and the other process is to calculate the coordinate transformation relationship of translation. These two processes complement each other, and these two processes are not limited to the above content.

[0068] After the scanner collects images, calibration board images in different poses are obtained. In the calibration board images of each pose, there are also differences in the features of the marking points, such as the shape and coordinate position of the same marking point. After detecting according to a certain image based on these differences, the image detection results in different poses are also different.

[0069] Step 204: Detect each fiducial point in the calibration board images with different poses according to at least two patterns, and obtain at least two groups of different fiducial point regions.

[0070] Each fiducial point in the calibration board image has its own pixel coordinate position region, and there are corresponding fiducial point features in the pixel coordinate position region of each fiducial point. After detecting these fiducial point features according to the corresponding images, multiple fiducial point regions corresponding to each fiducial point in the calibration board image are determined based on the detection results. The multiple fiducial point regions corresponding to each fiducial point are divided into corresponding different groups of fiducial point regions according to the different patterns for detection. The fiducial point regions in the same group are all the outlines of the same pattern, but the image attributes such as the position, side length, and area of the fiducial point regions in the same group are different. Whether each fiducial point region in the same group should be removed can be judged by at least one image attribute.

[0071] After some fiducial point regions in the same group are removed, the corresponding relationship between the fiducial point regions of different groups is determined based on the position of the fiducial point regions. Based on this corresponding relationship, the respective fiducial point regions corresponding to the fiducial point regions of different groups are determined respectively to judge whether the region centers of the fiducial point regions in each group are valid.

[0072] Step 206: Judge whether the region centers of the fiducial point regions in each group are valid based on the distances between the fiducial point regions in each group.

[0073] Regarding the distances between the fiducial point regions in each group, it can be the distances between the fiducial point regions in the same group and the distances between the fiducial point regions in different groups. When the distance between the fiducial point regions in the same group is less than the threshold of non-maximum suppression, the terminal judges that this part of the fiducial point regions in the same group is one of the non-maximum calculation regions of the adjacent point region and the duplicate point region, and performs non-maximum calculation on the non-maximum calculation region to remove the too adjacent points or duplicate points.

[0074] Regarding the distances between the fiducial point regions in different groups, the terminal calculates the distances between the corresponding fiducial point regions in different groups, compares this distance with the corresponding threshold, and determines whether the region center of the fiducial point regions in the group corresponding to a certain pattern is valid. For example: The fiducial point regions in the ellipse group have ellipse outlines, and the fiducial point regions in the quadrilateral group have quadrilateral outlines. When the distance between the corresponding fiducial point regions in the ellipse group and the quadrilateral group is less than the corresponding threshold, the region centers of the fiducial point regions in the ellipse group are respectively regarded as valid, while the region centers of the corresponding fiducial point regions in the quadrilateral group are all invalid.

[0075] After obtaining the center of each region, determine whether the number of region centers in the fiducial board image of each pose matches the number of fiducial points where the calibration board is attached. If they match, determine the center fiducial point corresponding to the corresponding pose based on the region center of each pose; if they do not match, perform abnormal processing on the number of coding points until the number of region centers in the fiducial board image of each pose matches the number of fiducial points where the calibration board is attached.

[0076] Among them, the abnormal processing of the number of region centers includes: if the number of region centers of a certain pose is less than the number of fiducial points where the calibration board is attached, adjust the pose and rescan until the abnormality disappears; if the number of region centers is greater than the number of fiducial points where the calibration board is attached, sequentially remove the points with the largest distance between the region center and the average pixel coordinates until the abnormality disappears.

[0077] Step 208: Generate fiducial point sequences for different poses based on each valid region center and the center fiducial points of each valid region center in different poses.

[0078] The center fiducial point is one of the region centers in a certain pose, and the center fiducial point is used to represent the center position fiducial point set on the fiducial board. Since the poses to which the calibration board images belong are different, the centroid position of each region center in the same calibration board image is not the center fiducial point, but the region center closest to the centroid position of each region center is the center fiducial point.

[0079] For the calculation process of the center fiducial points of different poses, it includes: the terminal performs averaging on the position data of each region center in a certain pose to obtain the average value of the region centers in that pose, and the average value of the region centers is the centroid position of the fiducial points in a certain pose; calculate the distance between the centroid position of this pose and each region center to obtain the calculation results between each region center and the centroid position, and based on this calculation result, select one of the region centers in this pose as the center fiducial point; among them, this calculation result represents the distance between the centroid position and the center fiducial point in this pose, which is less than the distance between the centroid position and any region center in this pose.

[0080] After obtaining the center fiducial point, determine the distance between each valid region center and the center fiducial point, so as to establish a corresponding relationship between each region center and the corresponding fiducial point. This corresponding relationship determines the sorting of each fiducial point in the calibration board image and constitutes a fiducial point sequence. Each fiducial point in the fiducial point sequence can also be encoded to facilitate better identification and call of the fiducial points.

[0081] In one embodiment, according to each valid regional center and the center landmark points of each valid regional center in different poses, a landmark point sequence in different poses is generated, including: performing averaging calculation based on the distance between the center of gravity positions with the same pose and the valid regional centers to obtain the center of gravity positions of the regional centers in different poses; based on the distance between the center of gravity positions with the same pose and the valid regional centers, searching for the center landmark points of each pose from each regional center; respectively determining the center landmark points of each regional center in different poses as the polar coordinate origins in each pose; based on each regional center and the polar coordinate origins in each pose, obtaining the positions of each regional center in the polar coordinate system of each pose; sorting the positions of each regional center in the polar coordinate system of each pose according to the angles of each regional center in the polar coordinate system to obtain the landmark point sequence in different poses.

[0082] After determining the polar coordinate origin, calculate the position where the regional center with the smallest adjacent angle is the starting landmark point, set the angle corresponding to the starting landmark point as the starting angle, and construct a polar coordinate system based on the polar coordinate origin and the starting landmark point. Although the angles of each regional center in the polar coordinate system are different, after rotation and tilt, because the number of each landmark point and its order in the polar coordinates are determined, and each landmark point is based on the starting landmark point and the polar coordinate origin, according to the angles of each regional center in the polar coordinate system, the relative positions of each landmark point in the landmark point sequence can still be known, that is, the landmark point sequence in the polar coordinate system has the characteristics of anti-rotation and anti-tilt.

[0083] In one embodiment, respectively determining the center landmark points of each regional center in different poses as the polar coordinate origins in each pose includes: respectively averaging the regional centers with the same pose to obtain the center of gravity positions of the regional centers in each pose; screening each regional center based on the distance interval between the center of gravity position of the regional centers with the same pose and the regional centers to obtain the center landmark points of each pose; respectively determining the center landmark points of each pose as the polar coordinate origins in each pose.

[0084] In one embodiment, sorting the positions of each regional center in the polar coordinate system of each pose according to the angles of each regional center in the polar coordinate system to obtain the landmark point sequence in different poses includes: calculating the initial polar coordinate angles between the center landmark point and each regional center, and then sorting according to the initial polar coordinate angles to obtain the initial landmark point sequence; calculating the difference between the angles of adjacent two points in the initial landmark point sequence, and determining the position with the smallest modulus of the angle difference as the target starting point; based on the position of the target starting point in the initial landmark point sequence, determining the target angles between the target starting point and each regional center in the polar coordinate system, and rearranging each regional center based on the target angles to obtain the target polar coordinate sequence.

[0085] Step 210: Calibrate the coordinate system transformation relationship between the robot and the scanner through a sequence of fiducial points with different poses.

[0086] In one embodiment, calibrating the coordinate system transformation relationship between the robot and the scanner through a sequence of fiducial points with different poses includes:

[0087] Perform averaging processing on the position coordinates of each fiducial point in the sequence of fiducial points with the same pose respectively to obtain the average value of the three-dimensional reconstruction coordinates for each pose. Calculate the rotation value and the first translation value according to the PNP algorithm for the average value of the three-dimensional reconstruction coordinates for each pose;

[0088] Perform sphere fitting on the average value of the three-dimensional reconstruction coordinates for each pose to obtain a fiducial point fitting sphere. Calculate the second translation value based on the center of the sphere of the fiducial point fitting sphere and the position average value corresponding to the sequence of fiducial points with a preset pose;

[0089] Calibrate the rotational coordinate system transformation relationship between the robot and the scanner through the rotation value, and calibrate the translational coordinate system transformation relationship between the robot and the scanner through the first translation value and the second translation value.

[0090] Thus, after obtaining the sequence of fiducial points with different poses, the coordinate system transformation relationship between the robot and the scanner can accurately calculate the corresponding coordinate system transformation relationship based on the relevant algorithms of PNP.

[0091] In one embodiment, the coordinate system transformation relationship includes a rotational transformation relationship and a translational transformation relationship. Calibrating the coordinate system transformation relationship between the robot and the scanner through a sequence of fiducial points with different poses includes: calibrating the rotational transformation relationship between the robot and the scanner through the position information of the fiducial points moving during the transformation of different poses, and calculating the first translation vector; calculating the second translation vector through the center of the sphere obtained by fitting the sequence of fiducial points with different poses and the sequence of fiducial points with a preset pose; combining the first translation vector and the second translation vector to calibrate the translational transformation relationship.

[0092] In one embodiment, the process of calculating the first translation vector specifically includes: keeping the rotation value of the tool coordinate system relative to the robot coordinate system unchanged, changing the position of the end effector multiple times, recording the origin coordinate value of the tool coordinate system and the average value of the three-dimensional reconstruction coordinates of the target fiducial points obtained by the scanner at each changed position, and using these two sets of coordinate values with the PNP algorithm to obtain the rotation value and the first translation vector between the robot coordinate system and the scanner coordinate system.

[0093] The steps for calculating the second translation vector include: during the process of calculating the first translation vector, recording the mean value of the three-dimensional reconstruction coordinates of the target fiducial points at the last state, and then, while keeping the position of the origin of the tool coordinate system unchanged at this time, rotating the tool coordinate system multiple times to change its pose. At each pose, the mean value of the three-dimensional reconstruction coordinates of the fiducial points is obtained by the scanner. The mean values obtained at the above-mentioned multiple poses are distributed on a spherical surface. Fit the center coordinates of the sphere, and combine with the mean value of the three-dimensional reconstruction coordinates of the fiducial points recorded above to obtain the second translation vector.

[0094] Finally, combine the calculated first translation vector and the second translation vector to obtain the translation conversion relationship. For example: First, after collecting 8 sets of fiducial point sequences with different poses, the rotation value conversion relationship and the first translation vector can be obtained by using PNP calculation. Next, fit at least 7 sets of fiducial point sequences to obtain the center of the fitted sphere. The coordinates of the center of the sphere and the average value corresponding to the first set of fiducial point sequences with 0 rotation times are used to obtain the second translation vector. Finally, add the negative of the second translation vector to the first translation vector calculated by the above PNP to obtain the translation value conversion relationship between the robot and the 3D scanner.

[0095] In the above robot hand-eye coordinate conversion method, for each fiducial point in the calibration board images with different poses, detect according to at least two patterns. Through this image geometric feature detection method, filter out the fiducial point regions; based on the distances between the fiducial point regions in each group, determine whether the fiducial points in the calibration board images are valid, and then generate fiducial point sequences with different poses according to the centers of each valid region and the central fiducial points of each valid region in different poses, so as to determine the positions of each fiducial point in the calibration board image through the fiducial point sequences, thereby converting the robot hand-eye calibration calculation into the calculation of the translation and rotation relationship between point pairs in three-dimensional space. The corresponding translation and rotation relationship can be calculated based on the PnP algorithm, without the need to calculate based on the change relationship between poses, and the calculation error is easy to control, which is convenient for application developers to debug.

[0096] In one embodiment, in step 204, for each fiducial point in the calibration board images with different poses, detect according to at least two patterns to obtain at least two different fiducial point regions, including: for each fiducial point in the calibration board image, perform edge extraction detection according to at least two different patterns respectively to obtain at least two different image contours. Filter the image contours in the contour length intervals of each group. Based on the filtered image contours in each group, obtain at least two different fiducial point regions.

[0097] Specifically, for each fiducial point in the calibration plate image, edge extraction detection is performed according to at least two different figures respectively using the Canny operator to obtain at least two groups of different image contours. The shorter and longer useless contours are removed from the at least two groups of different image contours to obtain the selected groups of image contours. The selected groups of image contours are subjected to corresponding calculations to obtain at least two groups of different fiducial point regions.

[0098] In one embodiment, the image contours include circular contours and polygonal contours. Based on the selected groups of image contours, at least two groups of different fiducial point regions are obtained, including: calculating the similarity of the circular contours based on the contour area and contour length of the selected circular contours to obtain the circular contour similarity; based on the circular contour similarity, selecting the circular fiducial point regions corresponding to each fiducial point from the found circular contours; fitting the polygonal contours in each found group to obtain a polygonal fitted image; and using the quadrilateral contour in the polygonal fitted image as the quadrilateral fiducial point region corresponding to each fiducial point.

[0099] Among them, after calculating the circular contour similarity, the circular fiducial point regions with similarity less than the similarity threshold are selected to obtain the circular fiducial point regions corresponding to each fiducial point. Among them, circular similarity = (4.0 * PI * contour area) / (contour perimeter * contour perimeter + 1e - 7), and the similarity threshold can be 0.8.

[0100] Correspondingly, in step 206, it is determined whether the region centers of the fiducial point regions in each group are valid, including: determining whether the region center of the circular fiducial point region is valid based on the distance between the circular fiducial point region and the quadrilateral fiducial point region of each fiducial point. Among them, since the fiducial points are circular, the region center of the circular fiducial point region may be valid, while the region centers of the quadrilateral fiducial point regions are all invalid to improve the corresponding accuracy.

[0101] In one embodiment, in step 206, it is determined whether the region centers of the fiducial point regions in each group are valid based on the distance between the fiducial point regions in each group, including: performing overlap detection on the fiducial point regions in each group respectively, removing the overlapping fiducial point regions in each group to obtain the fiducial point regions after removal in each group; comparing the distance between the fiducial point regions after removal in each group with the fiducial point adjacent threshold distance to obtain multiple adjacent point comparison results; and respectively determining whether the region centers in the fiducial point regions after removal in each group are valid based on each adjacent point comparison result.

[0102] During the process of performing overlapping detection on the landmark point regions in each group, the terminal respectively selects the landmark point regions in the same group based on the distances between the landmark point regions in each group, calculates the overlapping detection attributes corresponding to the selected landmark point regions in the same group, and performs detection on each landmark point region in the same group based on the overlapping detection attributes corresponding to the landmark point regions, so as to remove the overlapping landmark point regions in each group based on the detection results.

[0103] In one embodiment, performing overlapping detection on the landmark point regions in each group respectively and removing the overlapping landmark point regions in each group to obtain the landmark point regions after removal in each group, including: performing combined comparison on the landmark point regions in each group respectively; calculating the overlapping detection distance between the landmark point regions obtained by the comparison; when the overlapping detection distance meets the contour detection threshold, calculating the region side length and region area of each landmark point region in the combined comparison respectively; and removing the overlapping landmark point regions in each group based on the region side length and region area of each landmark point region in the combined comparison to obtain the landmark point regions after removal in each group.

[0104] The overlapping detection distance is the distance between any landmark point regions in the landmark point regions in the same group, and this distance is the central distance between the landmark point regions obtained by arbitrarily combining and calculating the landmark point regions in the same group respectively.

[0105] For example: After performing detection according to a quadrilateral, for the multiple quadrilateral landmark point regions in the obtained quadrilateral group, perform combined comparison on any two of the landmark point regions in each quadrilateral landmark point region respectively, calculate the overlapping detection distance between the two landmark point regions obtained by the combined comparison, and perform contour detection when the overlapping detection distance is less than the corresponding contour detection threshold. During the contour detection process, calculate the region side length and region area of each of the two landmark point regions in the combined comparison respectively, calculate the ratio between the product of the region side lengths and the region area, and when this ratio is within the effective range, compare the areas of the two quadrilaterals in the combination, select and remove the one with the larger area, and keep the smaller one to obtain the landmark point regions after removal in the quadrilateral group.

[0106] Therefore, the present invention screens fiducial points through the geometric feature relationships of two-dimensional images, can accurately identify and encode fiducial points in the form of a fiducial point sequence under the rotation and translation of the XYZ axes, combines sub-pixel processing to improve the reconstruction accuracy of the binocular 3D scanner, and uses the PnP algorithm to calculate point pairs for robot hand-eye calibration calculation. The calculation process is simple, the error is easy to control and optimize, which is convenient for application developers to debug. Compared with the traditional hand-eye calibration method, the traditional method needs to solve the equation AX = XB based on the change relationship of the pose. Since the difference between the upper and lower poses is used in the solution, the correct solution of this equation highly depends on the accuracy of the measurement data. When there are slight problems with the measurement data, the obtained result will have a large deviation from the actual true value. However, the calculation principle of the present invention is relatively intuitive and can better discover problems occurring in actual use. Even when there are robot movement errors, 3D scanner accuracy errors, ball center fitting errors, and errors in calculating rotation and translation during the measurement process, it can still ensure high accuracy.

[0107] It should be understood that although the steps in the flowcharts involved in the above embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0108] Based on the same inventive concept, the embodiments of the present application also provide a robot hand-eye coordinate conversion device for implementing the robot hand-eye coordinate conversion method involved above. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the robot hand-eye coordinate conversion device provided below can refer to the limitations on the robot hand-eye coordinate conversion method in the above text, and will not be repeated here.

[0109] In one embodiment, as Figure 5 shown, a robot hand-eye coordinate conversion device is provided, including: an image acquisition module 502, an edge detection module 504, a valid fiducial point determination module 506, a fiducial point sequence generation module 508, and a hand-eye calibration module 510, where:

[0110] The image acquisition module 502 is configured to acquire images of the end effector of the robot with a calibration board installed thereon through a scanner of the robot, so as to obtain calibration board images of the end effector in different poses;

[0111] The edge detection module 504 is configured to detect each fiducial point in the calibration board images in different poses according to at least two patterns, so as to obtain at least two groups of different fiducial point regions;

[0112] The valid fiducial point determination module 506 is configured to determine whether the region centers of the fiducial point regions in each group are valid based on the distances between the fiducial point regions in each group;

[0113] The fiducial point sequence generation module 508 is configured to generate fiducial point sequences in different poses according to each valid region center and the center fiducial points of each valid region center in different poses;

[0114] The hand-eye calibration module 510 is configured to calibrate the coordinate transformation relationship between the robot and the scanner through the fiducial point sequences in different poses.

[0115] In one embodiment, the edge detection module 504 includes:

[0116] The contour detection unit is configured to perform edge extraction detection on each fiducial point in the calibration board image according to at least two different patterns respectively, so as to obtain at least two groups of different image contours;

[0117] The length screening unit is configured to screen the image contours in the contour length intervals of each group;

[0118] The fiducial point region generation unit is configured to obtain the at least two groups of different fiducial point regions based on the image contours screened in each group.

[0119] In one embodiment, the fiducial point region generation unit includes:

[0120] The similarity calculation unit is configured to calculate the similarity of circular contours based on the contour area and contour length of the circular contours screened, so as to obtain the circular contour similarity;

[0121] The circular region screening unit is configured to select the circular fiducial point regions corresponding to each fiducial point from the found circular contours based on the circular contour similarity;

[0122] The polygon fitting unit is configured to fit the polygon contours in each group found, so as to obtain a polygon fitting image;

[0123] A quadrilateral region screening unit, configured to fit the quadrilateral contours in the polygon-fitted image as the quadrilateral landmark regions corresponding to the respective landmarks;

[0124] The valid landmark determination module 506 includes:

[0125] A center validity judgment unit, configured to determine whether the region center of the circular landmark region is valid based on the distance between the circular landmark region and the quadrilateral landmark region of each landmark.

[0126] In one embodiment, the valid landmark determination module 506 includes:

[0127] An overlapping region detection unit, configured to perform overlapping detection on the landmark regions in each group respectively, and remove the overlapping landmark regions in each group to obtain the landmark regions after removal in each group;

[0128] A neighboring point comparison unit, configured to compare the distances between the landmark regions after removal in each group with a landmark neighboring threshold distance to obtain a plurality of neighboring point comparison results;

[0129] A center validity judgment unit, configured to respectively judge whether the region centers in the landmark regions after removal in each group are valid based on each of the neighboring point comparison results.

[0130] In one embodiment, the overlapping region detection unit includes:

[0131] A region combination comparison subunit, configured to perform combination comparison on the landmark regions in each group respectively;

[0132] A first detection subunit, configured to calculate the overlapping detection distance between the landmark regions subjected to combination comparison;

[0133] A second detection subunit, configured to calculate the region side length and region area of each of the landmark regions subjected to combination comparison respectively when the overlapping detection distance meets the contour detection threshold;

[0134] An overlapping region removal subunit, configured to remove the overlapping landmark regions in each group based on the region side length and region area of each of the landmark regions subjected to combination comparison to obtain the landmark regions after removal in each group.

[0135] In one embodiment, the landmark sequence generation module 508 includes:

[0136] A centroid position calculation unit, configured to perform averaging calculation based on the valid region centers with the same pose to obtain the centroid position of the region centers with different poses;

[0137] A central landmark point determination unit, configured to find the central landmark point of each pose from the centers of the respective regions based on the distance between the center of gravity position with the same pose and the center of the valid region;

[0138] An origin generation unit, configured to respectively determine the central landmark points of the respective region centers in different poses as the polar coordinate origins in the respective poses;

[0139] A polar coordinate system construction unit, configured to obtain the positions of the respective region centers in the polar coordinate systems of the respective poses based on the respective region centers and the polar coordinate origins in the respective poses;

[0140] A landmark point sequence generation unit, configured to sort the positions of the respective region centers in the polar coordinate systems of the respective poses according to the angles of the respective region centers in the polar coordinate systems, to obtain landmark point sequences of different poses.

[0141] In one embodiment, the coordinate system conversion relationship includes a rotation conversion relationship and a translation conversion relationship. The hand-eye calibration module 510 includes:

[0142] A rotation relationship calibration unit, configured to calibrate the rotation conversion relationship between the robot and the scanner through the rotation vectors corresponding to the landmark point sequences during different pose conversions;

[0143] A first vector calculation unit, configured to obtain a first translation vector through the origin translation information of the landmark point sequences during different pose conversions;

[0144] A second vector calculation unit, configured to calculate through the center of the landmark point sphere obtained by fitting the landmark point sequences of different poses and the landmark point sequence of a preset pose to obtain a second translation vector;

[0145] A translation relationship calibration unit, configured to combine the first translation vector and the second translation vector to obtain the translation conversion relationship.

[0146] Each module in the above robot hand-eye coordinate conversion device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor in the computer device in the form of hardware or be independent of it, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above respective modules.

[0147] In one embodiment, a computer device is provided. The computer device can be a terminal, and its internal structure diagram can be as Figure 6As shown in the figure. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a robot hand-eye coordinate transformation method. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0148] Those skilled in the art can understand that Figure 6 the structure shown in the figure is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0149] In one embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0150] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0151] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0152] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions.

[0153] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memories can include read-only memory (ROM), magnetic tapes, floppy disks, flash memories, optical memories, high-density embedded non-volatile memories, resistive random access memories (ReRAM), magnetoresistive random access memories (MRAM), ferroelectric random access memories (FRAM), phase change memories (PCM), graphene memories, etc. Volatile memories can include random access memory (RAM) or external cache memories, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., and are not limited thereto. The processors involved in the embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., and are not limited thereto.

[0154] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0155] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation to the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A method for robot hand-eye coordinate transformation, characterized in that, The method includes: Performing image acquisition on the end effector of the robot equipped with a calibration plate through the scanner of the robot to obtain calibration plate images of the end effector in different poses; Performing edge extraction detection on each fiducial point in the calibration plate image according to at least two different patterns respectively to obtain at least two groups of different image contours; screening the image contours in the contour length intervals of each group; the image contours include circular contours and polygonal contours; Calculating the similarity of the circular contours based on the contour area and contour length of the screened circular contours to obtain the circular contour similarity; based on the circular contour similarity, selecting the circular fiducial point regions corresponding to each fiducial point from the screened circular contours; fitting the polygonal contours in each of the screened groups to obtain a polygonal fitted image; using the quadrilateral contour in the polygonal fitted image as the quadrilateral fiducial point region corresponding to each fiducial point; Determining whether the region center of the circular fiducial point region is valid based on the distance between the circular fiducial point region and the quadrilateral fiducial point region of each fiducial point; Generating fiducial point sequences in different poses based on each valid region center and the center fiducial points of each valid region center in different poses; each valid region center is the region center of the circular fiducial point region; the center fiducial point is one of the region centers in the pose of the calibration plate image, and the center fiducial point is used to represent the center position fiducial point set on the fiducial plate; Calibrating the coordinate transformation relationship between the robot and the scanner through the fiducial point sequences in different poses.

2. The method according to claim 1, wherein The method further includes: Performing overlapping detection on the fiducial point regions in each group respectively, and removing the overlapping fiducial point regions in each group to obtain the fiducial point regions after removal in each group; the overlapping detection is performed on the fiducial point regions in the same group respectively; Comparing the distances between the fiducial point regions after removal in each group with the fiducial point adjacent threshold distance to obtain multiple adjacent point comparison results; Based on each of the adjacent point comparison results, respectively determining whether the region centers in the fiducial point regions after removal in each group are valid.

3. The method according to claim 2, characterized in that, The performing overlapping detection on the fiducial point regions in each group respectively, and removing the overlapping fiducial point regions in each group to obtain the fiducial point regions after removal in each group includes: Performing combined comparison on the fiducial point regions in each group respectively; Calculating the overlapping detection distance between the combined and compared fiducial point regions; When the overlapping detection distance meets the contour detection threshold, respectively calculating the region side length and region area of each of the combined and compared fiducial point regions; Based on the region side length and region area of each of the combined and compared fiducial point regions, removing the overlapping fiducial point regions in each group to obtain the fiducial point regions after removal in each group.

4. The method according to claim 1, wherein The generating fiducial point sequences in different poses based on each valid region center and the center fiducial points of each valid region center in different poses includes: Performing averaging calculation based on the valid region centers with the same pose to obtain the centroid positions of the region centers in each pose; Find the central landmark points of each pose from the center points of the respective regions based on the distance between the center of gravity position with the same pose and the center of the effective region; Determine the central landmark points of the respective region centers in different poses as the origin points of the polar coordinate systems in the respective poses; Based on the center points of the respective regions and the origin points of the polar coordinate systems in the respective poses, obtain the positions of the center points of the respective regions in the polar coordinate systems of the respective poses; Sort the positions of the center points of the respective regions in the polar coordinate systems of the respective poses according to the angles of the center points of the respective regions in the polar coordinate systems, to obtain the landmark point sequences of different poses; 5. The method according to any one of claims 1 to 4, characterized in that, The coordinate system transformation relationship includes the transformation relationship of rotation and the transformation relationship of translation. Calibrating the coordinate system transformation relationship between the robot and the scanner through the landmark point sequences of different poses includes: Calibrate the rotation transformation relationship between the robot and the scanner through the position information of the movement of the landmark point sequences during the transformation of different poses; Obtain a first translation vector through the position information of the movement of the landmark point sequences during the transformation of different poses; Calculate a second translation vector through the center of the landmark ball fitted by the landmark point sequences of different poses and the landmark point sequence of a preset pose; Combine the first translation vector and the second translation vector to obtain the translation transformation relationship.

6. The method according to claim 5, characterized in that, The obtaining of the first translation vector through the position information of the movement of the landmark point sequences during the transformation of different poses includes: Under the premise of keeping the rotation value of the tool coordinate system relative to the robot coordinate system unchanged, change the position of the end effector multiple times; At each changed position, record the origin coordinate value of the tool coordinate system and the average value of the three-dimensional reconstruction coordinates of the target landmark points obtained by the scanner; Perform pose solution based on the origin coordinate value of the tool coordinate system and the average value of the three-dimensional reconstruction coordinates of the target landmark points to obtain the first translation vector.

7. A robot hand-eye coordinate conversion device, characterized in that, The device includes: An image acquisition module, configured to acquire images of the end effector of the robot with a calibration board installed thereon through a scanner of the robot, to obtain calibration board images of the end effector in different poses; An edge detection module, configured to perform edge extraction and detection on each landmark point in the calibration board image according to at least two different figures respectively, to obtain at least two groups of different image contours; screen the image contours in the contour length intervals of each group; the image contours include circular contours and polygon contours; calculate the similarity of the circular contours based on the contour area and contour length of the screened circular contours, to obtain the circular contour similarity; based on the circular contour similarity, select the circular landmark point regions corresponding to each landmark point from the screened circular contours; fit the polygon contours in each of the screened groups to obtain a polygon fitting image; use the quadrilateral contour in the polygon fitting image as the quadrilateral landmark point region corresponding to each landmark point; An effective landmark point determination module, configured to determine whether the region center of the circular landmark point region is effective based on the distance between the circular landmark point region and the quadrilateral landmark point region of each landmark point; A landmark point sequence generation module, configured to generate landmark point sequences in different poses according to each effective region center and each effective region center's center landmark point in different poses; each effective region center is the region center of the circular landmark point region; the center landmark point is one of the region centers in the pose of the calibration board image, and the center landmark point is used to represent the center position landmark point provided on the landmark board; A hand-eye calibration module, configured to calibrate the coordinate system conversion relationship between the robot and the scanner through the landmark point sequences in different poses.

8. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images