Online calibration method and system for industrial robot surface structured light stereo camera pose

Abstract

A method and system for online calibration of the posture of an industrial robot's surface structured light stereo camera. The calibration method includes placing the calibration structural features of a calibration cooperative target within the field of view of the structured light stereo camera below the industrial robot's first calibration pose; using the surface structured light stereo camera to collect three-dimensional point cloud data of the calibration cooperative target; identifying, segmenting, and calculating the calibration structural feature information of the calibration cooperative target in the first calibration pose, and automatically optimizing and adjusting the data of the remaining calibration poses; the industrial robot drives the surface structured light stereo camera to move to the remaining calibration poses, and the surface structured light stereo camera collects three-dimensional point cloud data of the calibration cooperative target in each pose; identifying, segmenting, and calculating the calibration structural feature information of the calibration cooperative target in the remaining calibration poses; and calculating the posture conversion information between the surface structured light stereo camera and the industrial robot. The present invention realizes online verification or calibration of the posture of the surface structured light stereo camera and the industrial robot, reducing operational complexity.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent industrial robots, and in particular relates to an online calibration method and system for an industrial robot's surface structured light stereo camera. Background Art

[0002] The development of intelligent industrial robots with depth perception, intelligent decision-making, and automated execution capabilities is a hot topic today. Compared to linear structured light stereo cameras, surface structured light stereo cameras can acquire 3D point cloud data of the target's exterior surface within a large field of view using a single frame of data in a static state. This effectively avoids the stitching errors introduced by linear structured light stereo cameras during the multi-pose motion of industrial robots, resulting in high overall accuracy and ease of use. Currently, surface structured light stereo cameras installed at the end of industrial robots are finding increasing application in industrial production.

[0003] The surface structured light stereo camera outputs three-dimensional point cloud data based on its own coordinate system. When controlling the operation of an industrial robot, its measurement data must be converted with high precision into data in the industrial robot coordinate system required by the industrial robot's end manipulator or other process execution units. This requires the intelligent industrial robot to calibrate the installation posture of the surface structured light stereo camera and the industrial robot before use to obtain the rotation and translation transformation matrix.

[0004] For pose calibration of surface structured light stereo cameras mounted on the end of industrial robots, which consist of one or more 2D industrial cameras, the industry currently largely uses a traditional calibration method that uses a coordinate transformation matrix between the 2D industrial camera and the industrial robot. This involves placing a planar cooperative target, such as a checkerboard, on the work surface and manually adjusting the cooperative target in different poses to complete the 2D industrial camera's intrinsic calibration. The robot then moves to multiple locations at different poses. The 2D industrial camera captures and processes images of the cooperative target at each location, extracting feature corners. Using the affine transformation principle, the coordinate transformation matrix between the 2D industrial camera and the industrial robot is calculated and used to replace the coordinate transformation matrix between the surface structured light stereo camera and the industrial robot. However, the relationship between the 2D coordinates in pixel space and the 3D coordinates in object space is not linear. Surface structured light stereo cameras typically require nonlinear revision of the initially calculated 3D point cloud data to obtain higher-precision 3D point cloud data. Consequently, this calibration method results in significant deviations from the 3D coordinates in real space when converting the surface structured light stereo camera coordinates to the industrial robot coordinates, resulting in low accuracy. Furthermore, this calibration method requires additional calibration fixtures and accessories. When calibrating the camera's intrinsic parameters, it's necessary to manually position the planar cooperative target in different postures, which is time-consuming and difficult to automate. It's also impossible to verify or calibrate the coordinate transformation matrix between the surface structured light stereo camera and the industrial robot online during use. Currently, the industry also uses stereo cooperative targets. One type involves installing a point-aligning fixture on the robot flange before installing the operating manipulator unit. The three-dimensional features captured by the surface structured light stereo camera are then aligned to establish a correspondence between the two coordinate systems. Another type involves designing stereo cooperative targets with significant three-dimensional features, such as spheres or spires. The surface structured light stereo camera collects and processes the three-dimensional point cloud data of the stereo cooperative target at each point. The general algorithm in the PCL point cloud library is used to extract the coordinates of the sphere center or spire feature corners in the three-dimensional camera coordinate system. Combined with the known coordinates of the sphere center or spire in the industrial robot coordinate system, the coordinate transformation matrix between the surface structured light stereo camera and the industrial robot can be solved. This calibration method uses a general algorithm to extract the center or top features of the sphere, which is heavily dependent on the quality of the 3D point cloud of the stereo cooperative target. It often requires manual removal of poor-quality 3D point cloud data and can only achieve offline calibration. It also requires additional calibration tooling accessories, is complex to operate, is time-consuming, and cannot perform online verification or calibration of the coordinate transformation matrix between the structured light stereo camera and the industrial robot during use. Summary of the Invention

[0005] The purpose of the present invention is to address the problems in the above-mentioned prior art and provide a method and system for online calibration of the posture of an industrial robot's surface structured light stereo camera, so as to realize online verification or calibration of the posture of the surface structured light stereo camera and the industrial robot during use, reduce the complexity of operation, save time, and improve accuracy.

[0006] In order to achieve the above object, the present invention has the following technical solutions:

[0007] An online calibration method for an industrial robot's surface structured light stereo camera pose, comprising:

[0008] After the calibration cooperation target is stably placed, the calibration structural features of the calibration cooperation target are within the field of view of the structured light stereo camera below the first calibration posture of the industrial robot; the calibration cooperation target is a polyhedral structural block, and the calibration structural features are characteristic shape structures processed on the polyhedral structural block;

[0009] The industrial robot drives the surface structured light stereo camera installed on the end manipulator to move to the first calibration pose. The surface structured light stereo camera collects 3D point cloud data of the calibration cooperation target; identifies, segments, and calculates the calibration structure feature information of the calibration cooperation target in the first calibration pose, performs feature matching, and automatically optimizes and adjusts the remaining calibration pose data;

[0010] The industrial robot drives the surface structured light stereo camera to move to the remaining calibration poses, and the surface structured light stereo camera collects the three-dimensional point cloud data of the calibration cooperation target at each pose; identifies, segments, and calculates the calibration structural feature information of the calibration cooperation targets at the remaining calibration poses;

[0011] The pose change information of the industrial robot at each calibration pose and the pose change information of the calibration cooperation target collected by the surface structured light stereo camera at each calibration pose are combined to calculate the pose conversion information of the surface structured light stereo camera and the industrial robot.

[0012] Preferably, the determination of the first calibration pose of the industrial robot comprises the following steps:

[0013] After placing the calibration cooperation target stably, control the industrial robot to move near the calibration cooperation target;

[0014] The surface structured light stereo camera collects a frame of three-dimensional point cloud data, identifies, segments, and calculates the calibration structural feature information of the calibration cooperation target, performs feature matching, and optimizes and adjusts the first calibration pose of the industrial robot.

[0015] Preferably, when the surface structured light stereo camera collects and calibrates three-dimensional point cloud data of the cooperative target, determining the acquisition parameters includes:

[0016] For the first time, the structured light stereo camera collects 3D point cloud data of the calibration cooperation target based on pre-set exposure time, diffuse reflection or multi-reflection parameters;

[0017] Combined with the performance of the structured light stereo camera, the system calculates and analyzes the level of 3D point cloud data that meets the calibration requirements, and automatically optimizes and adjusts the acquisition parameters.

[0018] According to the automatically optimized and adjusted acquisition parameters, the surface structured light stereo camera is controlled to collect 3D point cloud data of the calibration cooperation target multiple times to ensure that the 3D point cloud data meets the calibration requirements.

[0019] Preferably, identifying, segmenting, and calculating the calibration structural feature information of the first calibration pose calibration cooperation target includes:

[0020] Calculate the normal change of the three-dimensional point cloud data of the calibration cooperation target, and segment the structural feature data of the calibration cooperation target;

[0021] Extracting coding features from the structural feature data of the segmented and calibrated cooperative target and identifying feature coding;

[0022] Calculate the plane normal and distance information of the structural features of the calibration cooperation target,

[0023] Based on the calibration structure feature information, feature matching is performed and the remaining calibration poses are automatically optimized and adjusted, including the following steps:

[0024] The pose of the calibration cooperative target is placed within a certain range. The actual placement pose of the calibration cooperative target under the first calibration pose is calculated and matched with the theoretical placement pose to obtain pose change information.

[0025] According to the pose change information, the remaining calibration pose data are automatically optimized and adjusted.

[0026] Preferably, the step of causing the industrial robot to drive the surface structured light stereo camera to move to other calibration poses, and the surface structured light stereo camera to respectively collect three-dimensional point cloud data of the calibration cooperation target at each pose includes:

[0027] For the first time, the structured light stereo camera collects 3D point cloud data of the calibration cooperation target based on pre-set exposure time, diffuse reflection or multi-reflection parameters;

[0028] Combined with the performance of the structured light stereo camera, the system calculates and analyzes the level of 3D point cloud data that meets the calibration requirements, and automatically optimizes and adjusts the acquisition parameters.

[0029] According to the automatically optimized and adjusted acquisition parameters, the surface structured light stereo camera is controlled to collect 3D point cloud data of the calibration cooperation target multiple times to ensure that the 3D point cloud data meets the calibration requirements.

[0030] Preferably, the steps of identifying, segmenting, and calculating the calibration structural feature information of the remaining calibration pose and calibration cooperative targets include:

[0031] Calculate the normal change of the three-dimensional point cloud data of the calibration cooperation target, and segment the structural feature data of the calibration cooperation target;

[0032] Extracting coding features from the structural feature data of the segmented and calibrated cooperative target and identifying feature coding;

[0033] Calculate the plane normal and distance information of the structural features of the calibration cooperation target.

[0034] Preferably, the posture change information of the industrial robot at each calibration posture is calculated according to the following steps:

[0035] Read the parameters of each calibration pose of the industrial robot;

[0036] The parameters of each calibration posture of the industrial robot are calculated according to the manufacturer's coordinate system definition; the posture matrix corresponding to the posture parameters is set to Rr n , the pose matrix and translation matrix are Tr n , where r represents the industrial robot; the rotation change information of the two-to-two calibration poses of the industrial robots is calculated by the following formula:

[0037] R n1n2 =Rr n1 *Rr n2 -1

[0038] Where Rr n1n2 Indicates the rotation change from the n2th calibration pose to the n1th calibration pose;

[0039] The translation change information of the two calibration poses is calculated as follows:

[0040] Tr n1n2 =Tr n1 -Rr n1n2 * Tr n1

[0041] Where Tr n1n2 Indicates the translation change from the n2th calibration pose to the n1th calibration pose;

[0042] The pose change information of the calibration cooperative target collected by the surface structured light stereo camera of each calibration pose is calculated according to the following steps:

[0043] Read the plane normal and distance information of the structural features of the calibration cooperation target for each calibration pose;

[0044] Set the pose matrix corresponding to the pose parameters to Rc n , the pose matrix and translation matrix are Tc n , where c represents the surface structured light stereo camera; decomposition calculation of the surface structured light stereo camera and the calibration cooperative target pairwise calibration pose rotation change information, the calculation expression is as follows:

[0045] Rcn1n2 =Rc n1 *Rc n2 -1

[0046] Where Rc n1n2 Indicates the rotation change from the n2th calibration pose to the n1th calibration pose;

[0047] Calculate the translation change information of the two calibration poses. The calculation expression is as follows:

[0048] Tc n1n2 =Tc n1 -Rc n1n2 * Tc n1 ;

[0049] Where Tr n1n2 Indicates the translation change from the n2th calibration pose to the n1th calibration pose;

[0050] The pose conversion information between the surface structured light stereo camera and the industrial robot is calculated as follows:

[0051] The rotation vector Vr corresponding to the rotation change matrix of the pairwise calibration posture of the industrial robot n1n2 Composition matrix A; the rotation vector Vc corresponding to the rotation change matrix of the pairwise calibration pose of the calibration cooperation target collected by the surface structured light stereo camera n1n2 Composition B; the position rotation change information of the surface structured light stereo camera and the industrial robot is Rrc according to the following formula:

[0052] Rrc=B×A -1 ;

[0053] Calculate the translation change information Trc as follows:

[0054] Trc =[Rrc* Tr n1n2 -Tc n1n2 ]*[ Rc n1n2 -I] -1 , where I is the third-order identity matrix;

[0055] [Rrc, Trc] is the pose conversion information between the surface structured light stereo camera and the industrial robot.

[0056] An online calibration system for the surface structured light stereo camera pose of an industrial robot, comprising:

[0057] An initial detection module is configured to stably place a calibration cooperative target so that its calibration structural features are within the field of view of a structured light stereo camera below the first calibration posture of the industrial robot; the calibration cooperative target is a polyhedral structural block, and the calibration structural features are characteristic shape structures machined on the polyhedral structural block;

[0058] The first pose calibration module is used to enable the industrial robot to drive the surface structured light stereo camera installed on the end manipulator to move to the first calibration pose. The surface structured light stereo camera collects 3D point cloud data of the calibration cooperation target; identifies, segments, and calculates the calibration structure feature information of the calibration cooperation target in the first calibration pose, performs feature matching, and automatically optimizes and adjusts the remaining calibration pose data;

[0059] Other pose calibration modules are used to enable the industrial robot to drive the surface structured light stereo camera to move to other calibration poses. The surface structured light stereo camera collects 3D point cloud data of each pose calibration cooperation target respectively; identifies, segments, and calculates the calibration structural feature information of the other calibration pose calibration cooperation targets;

[0060] The pose conversion information calculation module is used to combine the pose change information of the industrial robot at each calibration pose and the pose change information of the calibrated cooperative target collected by the surface structured light stereo camera at each calibration pose, and calculate the pose conversion information between the surface structured light stereo camera and the industrial robot.

[0061] Preferably, the characteristic shape structure includes surface structure features and point structure features. The surface structure features are three adjacent surfaces, each surface has a hole structure, and the number of hole structures corresponds to the coding sequence 1, 2, and 3 respectively; the point structure features are the common vertices of the three adjacent surfaces; according to the normal change of the three-dimensional point cloud data of the calibration cooperation target, the calibration structure feature information of the calibration cooperation target is segmented; the plane normal and distance information are used in a 3×4 matrix C n Represents, where the subscript n represents the number of the calibration pose, which is a positive integer from 1 to not less than 3; the first row of the matrix represents the plane normal and distance information of the structural feature code 1; the second row of the matrix represents the plane normal and distance information of the structural feature code 2; the third row of the matrix represents the plane normal and distance information of the structural feature code 3.

[0062] Compared with the prior art, the present invention has at least the following beneficial effects:

[0063] By using a calibration cooperative target, the industrial robot is controlled to drive the surface structured light stereo camera to move to the calibration position and attitude control parameters to collect the three-dimensional point cloud data of the calibration cooperative target, identify, segment and calculate the calibration structural feature information of the calibration cooperative target, and calculate the posture conversion information of the surface array structured light stereo camera and the industrial robot, which has the significant advantage of a simple and fast calibration process.

[0064] During the execution of the calibration method of the present invention, the calibration cooperation target only needs to be placed stably so that the calibration structural features of the calibration cooperation target are within the field of view of the structured light stereo camera below the first calibration posture of the industrial robot. Then, the calibration structural feature information of the calibration cooperation target is identified, segmented, and calculated, and feature matching is performed. The first calibration posture and the remaining calibration postures of the industrial robot are optimized and adjusted, and the method has the function of adapting to the placement deviation of the calibration cooperation target.

[0065] By combining 3D point cloud data with the performance of structured light stereo cameras, the system calculates and analyzes the degree to which the 3D point cloud meets calibration requirements. This system automatically optimizes and adjusts acquisition parameters to ensure that the 3D point cloud meets calibration requirements. Combined with the ability to adapt to placement deviations in calibration partners, the system offers the significant advantage of enabling online calibration of the installed poses of structured light stereo cameras and industrial robots.

[0066] Furthermore, the calibration cooperation target of the present invention is a polyhedral structural block, and the calibration structural feature is a characteristic shape structure processed on the polyhedral structural block. The characteristic shape structure includes a surface structural feature and a point structural feature, wherein the surface structural feature is three adjacent surfaces, each surface has a hole structure, and the number of hole structures corresponds to the coding sequence numbers 1, 2, and 3 respectively; the point structural feature is the common vertex of the three adjacent surfaces; and the calibration structural feature information of the calibration cooperation target is segmented according to the normal change of the three-dimensional point cloud data of the calibration cooperation target. The present invention adopts a calibration cooperation target with a polyhedral structural feature, each of which contains a coding feature. By identifying, segmenting, and calculating the plane normal and distance information of the calibration structural feature, the posture change accuracy of the surface structure light stereo camera can be improved, the robustness of the calibration is improved, and the problem of inconsistent quality of three-dimensional point cloud data caused by changes in ambient light can be adapted. It has the significant advantage of high calibration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 Schematic diagram of the overall assembly structure of the robot surface structured light stereo camera online calibration device of the present invention;

[0068] Figure 2 A schematic diagram of the structure of the present invention for calibrating cooperation targets;

[0069] Figure 3 Flowchart of the online calibration method for the surface structured light stereo camera of an industrial robot according to the present invention;

[0070] In the attached figure: 1-industrial robot; 2-surface structured light stereo camera; 3-calibration cooperation target; 4-surface structured light stereo camera mounting part; 5-calibration platform; 6-field of view of the surface structured light stereo camera; 7-surface structural feature; 8-hole structure; 9-point structural feature. DETAILED DESCRIPTION

[0071] The present invention will be described in further detail below with reference to the accompanying drawings.

[0072] See also Figure 1 The overall assembly structure of an online calibration device for a robot surface structured light stereo camera according to an embodiment of the present invention includes a calibration cooperative target 3 disposed within the field of view 6 of the surface structured light stereo camera. The surface structured light stereo camera 2 is mounted on the end manipulator of an industrial robot 1. The calibration cooperative target 3 is a polyhedron structure block with a characteristic shape structure machined thereon. The industrial robot 1 drives the surface structured light stereo camera 2 to various calibration poses. The surface structured light stereo camera 2 captures images of the calibration cooperative target 3. Based on the pose changes of the calibration cooperative target 3 at each calibration pose of the industrial robot 1, pose conversion information between the surface structured light stereo camera and the industrial robot is obtained.

[0073] In an optional embodiment, the surface structured light stereo camera 2 is mounted on the industrial robot 1 via a surface structured light stereo camera mounting member 4. The surface structured light stereo camera mounting member 4 is designed according to the hardware layout of the production operation.

[0074] The calibration cooperation target 3 is stably installed on the calibration platform 5 .

[0075] The calibration cooperation target 3 is set within the field of view 6 of the structured light stereo camera below the first calibration posture of the industrial robot.

[0076] See also Figure 2 The characteristic shape structure processed on the polyhedral structural block of the present invention includes a surface structure feature 7 and a point structure feature 9. The surface structure feature 7 is three adjacent surfaces, each of which has a hole structure 8. The number of hole structures 8 corresponds to the coding numbers 1, 2, and 3 respectively; and the point structure feature 9 is the common vertex of the three adjacent surfaces.

[0077] Example 1

[0078] See also Figure 3 The method for online calibration of the industrial robot surface structured light stereo camera pose of the present invention comprises the following steps:

[0079] Step 1: Place the calibration cooperation target 3 stably and ensure that the calibration structural features of the calibration cooperation target 3 are within the field of view 6 of the surface structured light stereo camera of the industrial robot 1 in the first calibration posture.

[0080] Place the calibration cooperation target 3 stably, including the following steps:

[0081] 1a) The calibration cooperative target is placed stably and can be permanently fastened to the side of the work platform; it can also be temporarily fastened to the work platform during calibration. The purpose of fastening is to ensure the stability of the spatial position of the calibration cooperative target during the calibration process.

[0082] 1b) The calibration structure characteristics of the calibration cooperation goal are oriented upward;

[0083] In step 1b) of this embodiment, the calibration structural features of the calibration cooperative target 3 are oriented upward, and the directional deviation is no more than 5°.

[0084] The calibration structure feature of the calibration cooperation target 3 is placed within the field of view 6 of the surface structured light stereo camera of the industrial robot 1 in the first calibration posture, including the following steps:

[0085] 1c) Control industrial robot 1 to move near calibration cooperative target 3. The distance between the optimal viewing plane of the structured light stereo camera and the intersection of the three planes of the calibration cooperative target must not exceed 10% of the structured light stereo camera's field of view. Based on a 50mm one-way field of view, this distance deviation should not exceed 5mm.

[0086] Step 2: Issue a calibration command. The industrial robot 1 drives the surface structured light stereo camera 2 to move to the first calibration pose. The surface structured light stereo camera 2 controls the parameters and collects the 3D point cloud data of the calibration cooperative target 3.

[0087] The surface structured light stereo camera 2 collects and calibrates the three-dimensional point cloud data of the cooperative target 3, including the following steps:

[0088] 2a) The surface structured light stereo camera 2 first collects and calibrates the 3D point cloud data of the cooperative target 3 according to pre-set parameters such as exposure time, diffuse reflection or multiple reflection;

[0089] In step 2a) of this embodiment, the exposure time is preset to 6000 microseconds, and the diffuse reflection state is selected.

[0090] 2b) Based on the performance of the structured light stereo camera 2, calculate and analyze the level of 3D point cloud data that meets the calibration requirements, and automatically optimize and adjust the acquisition parameters when necessary;

[0091] In step 2b) of this embodiment, the flatness deviation value of the three-dimensional point cloud data does not exceed three times the Z-direction positioning accuracy of the surface structured light stereo camera. That is, if the Z-direction positioning accuracy of the surface structured light stereo camera is 0.035 mm, the flatness deviation of the three-dimensional point cloud data does not exceed 0.1 mm.

[0092] 2c) When necessary, the surface structured light stereo camera 2 is controlled to collect 3D point cloud data of the calibration cooperation target 3 multiple times according to the automatically optimized and adjusted acquisition parameters to ensure that the 3D point cloud data meets the calibration requirements.

[0093] In step 2c) of this embodiment, the acquisition parameters are optimized and adjusted based on the grayscale values ​​of the grayscale image of the surface structured light stereo camera 2 and the amount of 3D point cloud data. The grayscale values ​​of the grayscale image are maintained within the range of 100 to 230, and the amount of 3D point cloud data is maintained at more than 60%.

[0094] Step 3: Identify, segment, and calculate the calibration structural feature information of the calibration cooperative target 3, perform feature matching, and automatically optimize and adjust the remaining calibration pose data.

[0095] In this embodiment, the process of identifying, segmenting, and calculating the calibration structural feature information includes the following steps:

[0096] 3a) Calculate the normal change of the 3D point cloud data of the calibration cooperative target 3, and segment the structural feature data of the calibration cooperative target 3;

[0097] In step 3a) of this embodiment, point clouds with normal direction variations greater than 30 degrees are divided into point cloud data with different structural features.

[0098] 3b) extracting the coding features from the structural feature data of the segmented and calibrated cooperative target 3 and identifying the feature coding;

[0099] In step 3b) of this embodiment, the number of codes in the structural feature data is the same as the number of hole structures, which are recorded as codes 1, 2, and 3.

[0100] 3c) Calculate the plane normal and distance information of the structural features of the calibration cooperation target;

[0101] In step 3c) of this embodiment, the plane normal and distance information are represented by a 3×4 matrix C n , where subscript n is the number of the calibration pose, a positive integer from 1 to not less than 3. The first row of the matrix represents the plane normal and distance information of structural feature code 1. The second row of the matrix represents the plane normal and distance information of structural feature code 2. The third row of the matrix represents the plane normal and distance information of structural feature code 3.

[0102] In this embodiment, the method of performing feature matching based on the calibration structure feature information and automatically optimizing and adjusting the remaining calibration poses includes the following steps:

[0103] 3d) The pose of the manually placed calibration cooperative target 3 is within a certain range. To ensure that the surface structured light stereo camera 2 at each calibration pose can capture the calibration structural feature information of the calibration cooperative target 3, the actual placement pose of the calibration cooperative target 3 under the first calibration pose is calculated and matched with the theoretical placement pose to obtain pose change information;

[0104] 3e) The theoretical placement pose under the first calibration pose is C0, and the actual placement pose is C0'. The pose change is represented by the feature matching matrix, RT0 = C0' × C0 -1 . C0 -1 is the inverse matrix of the theoretical placement pose, and the remaining calibration pose data are left-multiplied by the feature matching matrix RT0 to obtain the optimized and adjusted calibration pose data.

[0105] In step 4, the industrial robot 1 drives the surface structured light stereo camera 2 to move to other calibration poses, and the surface structured light stereo camera 2 controls the parameters to collect the three-dimensional point cloud data of the calibration cooperation target in each pose.

[0106] In the remaining calibration poses, the acquisition parameters of the calibration cooperative target acquired by the surface structured light stereo camera 2 are determined, including the following steps similar to 2a) to 2c):

[0107] Step 5: Identify, segment, and calculate the calibration structure feature information of the remaining calibration pose and calibration cooperative targets.

[0108] In this embodiment, the process of identifying, segmenting, and calculating the calibration structural feature information includes the following steps:

[0109] 5a) Calculate the normal change of the 3D point cloud data of the calibration cooperative target 3, and segment the structural feature data of the calibration cooperative target 3;

[0110] In step 5a) of this embodiment, point clouds with a normal direction variation greater than 30 degrees are divided into point cloud data with different structural features.

[0111] 5b) extracting the coded features from the structural feature data of the segmented and calibrated cooperative target and identifying the feature codes;

[0112] In step 5b) of this embodiment, the number of codes in the structural feature data is the same as the number of hole structures, which are recorded as codes 1, 2, and 3.

[0113] 5c) Calculate the plane normal and distance information of the structural features of the calibration cooperation target.

[0114] In step 5c) of this embodiment, the plane normal and distance information are represented by a 3×4 matrix C n , where subscript n is the number of the calibration pose, a positive integer from 1 to not less than 3. The first row of the matrix represents the plane normal and distance information of structural feature code 1. The second row of the matrix represents the plane normal and distance information of structural feature code 2. The third row of the matrix represents the plane normal and distance information of structural feature code 3.

[0115] Step 6: Combine the pose change information of the industrial robot 1 at each calibration pose and the pose change information of the calibration cooperation target collected by the surface array structured light stereo camera 2 at each calibration pose to calculate the pose conversion information of the surface array structured light stereo camera 2 and the industrial robot 1.

[0116] In this embodiment, the steps for calculating the posture change information of the industrial robot at each calibration posture are as follows:

[0117] 6a) Read the parameters of each calibration pose of the industrial robot 1.

[0118] 6b) The parameters of each calibration pose of the industrial robot 1 are calculated according to the coordinate system definition of the manufacturer of the industrial robot 1. Here, the pose matrix corresponding to the pose parameters is set to Rr n , the pose matrix and translation matrix are Tr n , where r represents the industrial robot. Decompose and calculate the rotation change information of the two-way calibration pose of the industrial robot. The calculation formula is:

[0119] R n1n2 =Rr n1 *Rr n2 -1

[0120] In the formula, Rr n1n2 Indicates the rotation change from the n2th calibration pose to the n1th calibration pose.

[0121] Calculate the translation change information of the two calibration poses. The calculation formula is:

[0122] Tr n1n2 =Tr n1 -Rr n1n2 * Tr n1 .

[0123] In the formula, Tr n1n2 Indicates the translation change from the n2th calibration pose to the n1th calibration pose.

[0124] In this embodiment, the method for calculating each calibration pose by the surface structured light stereo camera 2 to collect pose change information of the calibration cooperative target 3 is as follows:

[0125] 6c) Reading the plane normal and distance information of the structural features of the calibration cooperation target for each calibration pose.

[0126] 6d) The pose of each calibration pose of the calibration cooperative target 3 is defined as the plane normal and distance information set with the encoded structural features of the calibration cooperative target 3. Here, the pose matrix corresponding to the pose parameters is set to Rc n , the pose matrix and translation matrix are Tc n, where c represents the surface structured light stereo camera. Decomposition calculation: The surface structured light stereo camera 2 collects the pairwise calibration pose rotation change information of the calibration cooperation target, and the calculation formula is:

[0127] Rc n1n2 =Rc n1 *Rc n2 -1

[0128] In the formula, Rc n1n2 Indicates the rotation change from the n2th calibration pose to the n1th calibration pose.

[0129] Calculate the translation change information of the two calibration poses. The calculation formula is:

[0130] Tc n1n2 =Tc n1 -Rc n1n2 * Tc n1 .

[0131] In the formula, Tr n1n2 Indicates the translation change from the n2th calibration pose to the n1th calibration pose.

[0132] The method for calculating the pose conversion information between the surface structured light stereo camera 2 and the industrial robot 1 is as follows:

[0133] 6e) The rotation vector Vr corresponding to the rotation change matrix of the pairwise calibration pose of industrial robot 1 n1n2 Composition matrix A; the rotation vector Vc corresponding to the pairwise calibration pose rotation change matrix of the calibration cooperation target 3 captured by the surface structured light stereo camera 2 n1n2 Component B: The position rotation change information of the surface structured light stereo camera 2 and the industrial robot 1 is Rrc, and the calculation formula is:

[0134] Rrc=B×A -1 .

[0135] 6f) Translation change information Trc, calculated as:

[0136] Trc =[Rrc* Tr n1n2 -Tc n1n2 ]*[ Rc n1n2 -I] -1 , where I is the third-order identity matrix.

[0137] 6h) [Rrc, Trc] is the pose conversion information between the surface structured light stereo camera 2 and the industrial robot 1.

[0138] Example 2

[0139] like Figure 2As shown, the calibration cooperation target is three planar structural features. The calibration cooperation target is installed on the working platform, and the calibration structural features of the calibration cooperation target face upward.

[0140] The industrial robot uses a KUKA brand KR70 R2100 six-axis industrial robot arm, and the surface structured light stereo camera is an LMI brand 3210 model binocular snapshot sensor.

[0141] Control the motion of industrial robots and calibrate the structural features of cooperative targets within the field of view of a structured light stereo camera.

[0142] After the calibration command is issued, the industrial robot drives the surface structured light stereo camera to move to the first calibration pose. The preset acquisition parameters of the surface structured light stereo camera are set to 6000 and diffuse reflection. After automatic optimization and adjustment, the exposure parameters are 3000 and diffuse reflection.

[0143] Set 5 calibration poses, and the optimized pose parameters are shown in Table 1:

[0144] Table 1 Industrial robot posture parameters

[0145]

[0146] The calculation results of structural characteristic data are shown in Table 2:

[0147] Table 2 Structural characteristics data

[0148]

[0149] The industrial robot's posture matrix Rrn and translation matrix Trn are shown in Table 3:

[0150] Table 3 Industrial robot pose matrix and translation matrix

[0151]

[0152] The pose matrix Rcn and translation matrix Tcn of the surface structured light stereo camera are shown in Table 4:

[0153] Table 4. Pose matrix and translation matrix of surface structured light stereo camera

[0154]

[0155] Using Rrc=B×A -1 and Trc =[Rrc*Tr n1n2 -Tc n1n2 ]*[ Rc n1n2 -I] -1 The pose conversion information [Rrc, Trc] between the surface structured light stereo camera and the industrial machine is calculated as:

[0156]

[0157] In summary, the present invention can solve the problems that existing calibration methods require complex operation procedures, are time-consuming, have low precision, and cannot verify or calibrate the posture of the surface structured light stereo camera and the industrial robot online during use.

[0158] Another embodiment of the present invention further provides an online calibration system for an industrial robot surface structured light stereo camera, comprising:

[0159] An initial detection module is configured to stably place a calibration cooperative target so that its calibration structural features are within the field of view of a structured light stereo camera below the first calibration posture of the industrial robot; the calibration cooperative target is a polyhedral structural block, and the calibration structural features are characteristic shape structures machined on the polyhedral structural block;

[0160] The first pose calibration module is used to enable the industrial robot to drive the surface structured light stereo camera installed on the end manipulator to move to the first calibration pose. The surface structured light stereo camera collects 3D point cloud data of the calibration cooperation target; identifies, segments, and calculates the calibration structure feature information of the calibration cooperation target in the first calibration pose, performs feature matching, and automatically optimizes and adjusts the remaining calibration pose data;

[0161] Other pose calibration modules are used to enable the industrial robot to drive the surface structured light stereo camera to move to other calibration poses. The surface structured light stereo camera collects 3D point cloud data of each pose calibration cooperation target respectively; identifies, segments, and calculates the calibration structural feature information of the other calibration pose calibration cooperation targets;

[0162] The pose conversion information calculation module is used to combine the pose change information of the industrial robot at each calibration pose and the pose change information of the calibrated cooperative target collected by the surface structured light stereo camera at each calibration pose, and calculate the pose conversion information between the surface structured light stereo camera and the industrial robot.

[0163] It should be noted that the information interaction, execution process, etc. between the above-mentioned module units are based on the same concept as the method embodiment. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0164] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0165] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0166] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0167] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. An online calibration method for the pose of an industrial robot surface structured light stereo camera, characterized in that: include: After the calibration cooperation target (3) is placed stably, the calibration structural features of the calibration cooperation target (3) are within the field of view (6) of the structured light stereo camera below the first calibration posture of the industrial robot (1); the calibration cooperation target (3) is a polyhedral structural block, and the calibration structural features are characteristic shape structures processed on the polyhedral structural block; The industrial robot (1) drives the surface structured light stereo camera (2) installed on the end manipulator to move to the first calibration posture, and the surface structured light stereo camera (2) collects three-dimensional point cloud data of the calibration cooperation target (3); identifies, segments, and calculates calibration structure feature information of the calibration cooperation target (3) in the first calibration posture, performs feature matching on the actual placement posture of the calibration cooperation target (3) in the first calibration posture and the theoretical placement posture, and obtains posture change information; according to the posture change information, automatically optimizes and adjusts the remaining calibration posture data; The industrial robot (1) drives the surface structured light stereo camera (2) to move to other calibration poses, and the surface structured light stereo camera (2) respectively collects three-dimensional point cloud data of each pose calibration cooperation target (3); identifies, segments, and calculates calibration structural feature information of the other calibration pose calibration cooperation targets (3); The pose change information of the industrial robot (1) at each calibration pose and the pose change information of the calibration cooperation target (3) collected by the surface structured light stereo camera (2) at each calibration pose are combined to calculate the pose conversion information of the surface structured light stereo camera (2) and the industrial robot (1).

2. The method for online calibration of the industrial robot surface structured light stereo camera pose according to claim 1, characterized in that: Determining the first calibration pose of the industrial robot (1) comprises the following steps: After the calibration cooperation target (3) is placed stably, the industrial robot (1) is controlled to move to the vicinity of the calibration cooperation target (3); The surface structured light stereo camera (2) collects a frame of three-dimensional point cloud data, identifies, segments, and calculates the calibration structure feature information of the calibration cooperation target (3), performs feature matching, and optimizes and adjusts the first calibration posture of the industrial robot (1).

3. The method for online calibration of the industrial robot surface structured light stereo camera pose according to claim 1, characterized in that: The determination of acquisition parameters when the surface structured light stereo camera (2) acquires the three-dimensional point cloud data of the calibration cooperation target (3) includes: The surface structured light stereo camera (2) collects three-dimensional point cloud data of the calibration cooperation target (3) for the first time according to the preset exposure time, diffuse reflection or multi-reflection parameters; Combined with the performance of the surface structured light stereo camera (2), the level of the three-dimensional point cloud data meeting the calibration requirements is calculated and analyzed, and the acquisition parameters are automatically optimized and adjusted; According to the acquisition parameters adjusted after automatic optimization, the surface structured light stereo camera (2) is controlled to acquire the three-dimensional point cloud data of the calibration cooperation target (3) multiple times to ensure that the three-dimensional point cloud data meets the calibration requirements.

4. The method for online calibration of the industrial robot surface structured light stereo camera pose according to claim 3, characterized in that: Identify, segment, and calculate the calibration structural feature information of the first calibration pose calibration cooperative target (3) including: Calculate the normal change of the three-dimensional point cloud data of the calibration cooperative target (3), and segment the structural feature data of the calibration cooperative target (3); Extracting coding features from the structural feature data of the segmented and calibrated cooperative target (3) and identifying the feature coding; Calculate the plane normal and distance information of the structural features of the calibration cooperation target (3), Based on the calibration structure feature information, feature matching is performed and the remaining calibration poses are automatically optimized and adjusted, including the following steps: The position and posture of the calibration cooperation target (3) is placed within a certain range, and the actual placement posture of the calibration cooperation target (3) under the first calibration posture is calculated and matched with the theoretical placement posture to obtain posture change information; According to the pose change information, the remaining calibration pose data are automatically optimized and adjusted.

5. The method for online calibration of the industrial robot surface structured light stereo camera pose according to claim 1, characterized in that: The steps of causing the industrial robot (1) to drive the surface structured light stereo camera (2) to move to other calibration postures, and the surface structured light stereo camera (2) to respectively collect three-dimensional point cloud data of the calibration cooperation target (3) at each posture include: The surface structured light stereo camera (2) collects three-dimensional point cloud data of the calibration cooperation target (3) for the first time according to the preset exposure time, diffuse reflection or multi-reflection parameters; Combined with the performance of the surface structured light stereo camera (2), the level of the three-dimensional point cloud data meeting the calibration requirements is calculated and analyzed, and the acquisition parameters are automatically optimized and adjusted; According to the acquisition parameters adjusted after automatic optimization, the surface structured light stereo camera (2) is controlled to acquire the three-dimensional point cloud data of the calibration cooperation target (3) multiple times to ensure that the three-dimensional point cloud data meets the calibration requirements.

6. The method for online calibration of the industrial robot surface structured light stereo camera pose according to claim 5, characterized in that: The steps of identifying, segmenting, and calculating the calibration structural feature information of the remaining calibration pose and calibration cooperative targets (3) include: Calculate the normal change of the three-dimensional point cloud data of the calibration cooperative target (3), and segment the structural feature data of the calibration cooperative target (3); Extracting coding features from the structural feature data of the segmented and calibrated cooperative target (3) and identifying the feature coding; Calculate the plane normal and distance information of the structural features of the calibration cooperation target (3).

7. The method for online calibration of the industrial robot surface structured light stereo camera pose according to claim 1, characterized in that: The posture change information of the industrial robot (1) at each calibration posture is calculated according to the following steps: Read the parameters of each calibration pose of the industrial robot (1); Industrial robot (1) The parameters of each calibration posture are calculated according to the manufacturer's coordinate system definition; the posture matrix corresponding to the posture parameters is set to Rr n , the pose matrix and translation matrix are Tr n , where r represents an industrial robot; Decompose and calculate the rotation change information of the two-to-two calibration postures of industrial robots according to the following formula: Rr n1n2 =Rr n1 *Rr n2 -1 Where Rr n1n2 Indicates the rotation change from the n2th calibration pose to the n1th calibration pose; The translation change information of the two calibration poses is calculated as follows: Tr n1n2 =Tr n1 - Rr n1n2 * Tr n1 Where Tr n1n2 Indicates the translation change from the n2th calibration pose to the n1th calibration pose; The pose change information of the calibration cooperation target (3) collected by the surface structured light stereo camera (2) of each calibration pose is calculated according to the following steps: Read the plane normal and distance information of the structural features of the calibration cooperation target (3) at each calibration pose; Set the pose matrix corresponding to the pose parameters to Rc n , the pose matrix and translation matrix are Tc n , where c represents the surface structured light stereo camera (2); decomposing and calculating the pairwise calibration pose rotation change information of the surface structured light stereo camera (2) and the calibration cooperative target (3), the calculation expression is as follows: RC n1n2 =Rc n1 *Rc n2 -1 Where Rc n1n2 Indicates the rotation change from the n2th calibration pose to the n1th calibration pose; Calculate the translation change information of the two calibration poses. The calculation expression is as follows: Tc n1n2 =Tc n1 - Rc n1n2 * Tc n1 ; Where Tr n1n2 Indicates the translation change from the n2th calibration pose to the n1th calibration pose; The posture conversion information of the surface structured light stereo camera (2) and the industrial robot (1) is calculated according to the following steps: The rotation vector Vr corresponding to the pairwise calibration pose rotation change matrix of the industrial robot (1) n1n2 The matrix A is composed of the rotation vector Vc corresponding to the rotation change matrix of the pairwise calibration pose of the calibration cooperation target (3) collected by the surface structured light stereo camera (2). n1n2 Composition B; the position rotation change information of the surface structured light stereo camera (2) and the industrial robot (1) is Rrc: Rrc=B×A -1 ; Calculate the translation change information Trc as follows: Trc =[Rrc* Tr n1n2 -Tc n1n2 ]*[ Rc n1n2 -I] -1 , where I is the third-order identity matrix; [Rrc, Trc] is the pose conversion information between the surface structured light stereo camera and the industrial robot.

8. An online calibration system for the surface structured light stereo camera pose of an industrial robot, characterized by: include: An initial detection module is used to place the calibration cooperation target (3) stably so that the calibration structural features of the calibration cooperation target (3) are within the field of view (6) of the structured light stereo camera below the first calibration posture of the industrial robot (1); the calibration cooperation target (3) is a polyhedral structural block, and the calibration structural features are characteristic shape structures processed on the polyhedral structural block; A first pose calibration module is used to enable the industrial robot (1) to drive the surface structured light stereo camera (2) installed on the end manipulator to move to the first calibration pose, and the surface structured light stereo camera (2) collects three-dimensional point cloud data of the calibration cooperation target (3); identifies, segments, and calculates calibration structure feature information of the calibration cooperation target (3) in the first calibration pose, performs feature matching on the actual placement pose of the calibration cooperation target (3) in the first calibration pose and the theoretical placement pose, and obtains pose change information; and automatically optimizes and adjusts the remaining calibration pose data according to the pose change information; Other posture calibration modules are used to enable the industrial robot (1) to drive the surface structured light stereo camera (2) to move to other calibration postures, and the surface structured light stereo camera (2) respectively collects three-dimensional point cloud data of each posture calibration cooperation target (3); recognizes, segments, and calculates calibration structural feature information of the other calibration posture calibration cooperation targets (3); The posture conversion information calculation module is used to combine the posture change information of the industrial robot (1) at each calibration posture and the posture change information of the calibration cooperation target (3) collected by the surface structured light stereo camera (2) at each calibration posture, and calculate the posture conversion information of the surface structured light stereo camera (2) and the industrial robot (1).

9. The online calibration system for the industrial robot surface structured light stereo camera pose according to claim 8, characterized in that: The characteristic shape structure includes a surface structure feature (7) and a point structure feature (9), wherein the surface structure feature (7) is three adjacent surfaces, each surface having a hole structure (8), and the number of hole structures (8) corresponds to the coding sequence 1, 2, and 3 respectively; the point structure feature (9) is a common vertex of the three adjacent surfaces; according to the normal change of the three-dimensional point cloud data of the calibration cooperation target (3), the calibration structure feature information of the calibration cooperation target (3) is segmented; the plane normal and distance information are converted into the matrix C of 3×4. n Represents, where the subscript n represents the number of the calibration pose, which is a positive integer from 1 to not less than 3; the first row of the matrix represents the plane normal and distance information of the structural feature code 1; the second row of the matrix represents the plane normal and distance information of the structural feature code 2; the third row of the matrix represents the plane normal and distance information of the structural feature code 3.

Images