Spatial pose recognition method and system for tire assembly industrial robot unstacking

Abstract

The invention belongs to the field of robot target workpiece space pose detection, and particularly discloses a space pose recognition method and system for tire assembly industrial robot unstacking. Comprising the following steps: collecting material rack tire assembly point cloud data; measuring calibration plate image data under n poses at the tail end of the robot so as to calibrate the relation between the coordinate system of the point cloud camera and the coordinate system of the flange at the tail end of the robot; preprocessing the point cloud data and then performing normal vector filtering processing to determine the circular boundary of the tire assembly; obtaining a boundary point cloud of the tire assembly through boundary detection, and fitting a three-dimensional circle by adopting an RANSAC algorithm based on the boundary point cloud to obtain a space pose of the tire assembly; and judging whether the spatial pose is correct or not. By integrating the advanced visual recognition technology and the pose estimation algorithm, the tire assembly can be quickly and accurately recognized and positioned, and therefore the automation level of the industrial robot in the unstacking operation is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of robot target workpiece spatial posture detection, and more specifically, relates to a spatial posture recognition method and system for depalletizing tire assembly industrial robots. Background Art

[0002] As new energy vehicles usher in vigorous development, the automobile manufacturing industry is facing its automation transformation and upgrading, and related sub-industries such as tire and wheel assembly are also facing problems such as production capacity, efficiency and safety. Among them, the production line scheduling task for tire assemblies is located in the upstream of the entire automobile industry chain, and its efficiency and quality directly affect the production rhythm and cost control of vehicle manufacturing. At present, the production scheduling of tire assembly lines still faces many challenges, such as the flexibility requirements under the multi-variety small batch production mode, the lack of automation upgrades of production equipment, and safety hazards in the process flow. Based on this, improving the efficiency and reliability of tire assembly scheduling has become a practical need of the automobile manufacturing industry.

[0003] The traditional method of the current production scheduling process is still mainly manual destacking, which is done by manually moving the tire assembly, assisted by truss clamps. During the production process, scheduling errors are prone to occur, leading to tire assembly problems, which in turn cause consumer dissatisfaction and subsequent driving safety issues; at the same time, because the tire assembly has a heavy load, manual handling is prone to cause work injuries, which hinders the development of its industry efficiency and makes it difficult to meet its industrial automation upgrade needs.

[0004] In addition, in order to address the shortcomings of manual depalletizing during production scheduling, automated equipment represented by heavy-duty industrial robots is the general trend of development in the industry. For the detection and grasping tasks of the target workpiece (i.e., tire assembly), the traditional teaching method cannot be implemented due to the floating position of the shelf on which the tire assembly is loaded and the floating position of the target workpiece. The detection method based on two-dimensional images is difficult to obtain its precise depth value, which can easily cause interference and collision problems. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the present invention provides a spatial posture recognition method and system for depalletizing tire assemblies by industrial robots, which collects the point cloud data of the tire assembly from viewpoint to viewpoint by loading a point cloud camera on the end fixture of the industrial robot, and uses point cloud related preprocessing algorithms such as voxel filtering and model fitting algorithms such as normal vector filtering and RANSAC three-dimensional circle fitting, finally realizing posture recognition and depalletizing and loading of the tire assembly. The present invention can be used in automobile assembly line systems to load and schedule assembled tire assemblies, improve the automation level of related production lines, enhance the robustness of the automobile assembly field, and effectively solve many deficiencies such as the efficiency and stability of traditional manual handling, and currently has great industrial application value.

[0006] To achieve the above object, according to one aspect of the present invention, a spatial posture recognition method for depalletizing a tire assembly industrial robot is proposed, comprising the following steps:

[0007] Step 1: Collect the point cloud data of the tire assembly of the rack viewpoint by viewpoint according to the configuration file;

[0008] Step 2: Measure the calibration plate image data at n positions of the robot end to calibrate the relationship between the point cloud camera coordinate system and the robot end flange coordinate system n≥3;

[0009] Step 3: Based on the relationship After preprocessing the point cloud data, normal vector filtering is performed to determine the circular boundary of the tire assembly;

[0010] Step 4: Obtain the boundary point cloud of the tire assembly through boundary detection, and fit a three-dimensional circle based on the boundary point cloud using the RANSAC algorithm to obtain the spatial pose of the tire assembly;

[0011] Step five, according to the position of the tire assembly rack and the distribution pattern of the tire assembly, calculate the estimated posture of the tire assembly, and determine whether the spatial posture is correct according to the error between the estimated posture and the spatial posture. If so, output the result, otherwise return to step three.

[0012] As further preferred, step one further comprises the following steps:

[0013] A configuration file is used to record multiple viewpoint positions of each tire layer, and the viewpoints of the current layer are traversed to collect the complete point cloud data of the current layer.

[0014] As further preferred, step 2 comprises the following steps:

[0015] (21) Traverse the n end flange positions of the robot, record the n positions, measure the image data of the calibration plate, and perform grayscale processing;

[0016] (22) Perform Gaussian blur processing and binarization processing on the grayscale image of the calibration plate;

[0017] (23) Calculate the robot hand-eye relationship based on the calibration plate image after binarization processing and the camera internal parameter matrix and distortion coefficient to obtain the relationship between the camera coordinate system and the robot end flange coordinate system

[0018] As a further preferred embodiment, in step 3, the normal vector filtering process after preprocessing the point cloud data comprises the following steps:

[0019] (31) Based on the robot hand-eye relationship Unify the multiple point clouds obtained in step 1 into the robot base coordinate system to obtain complete point cloud data;

[0020] (32) Based on the XYZ coordinate range of the rack spatial position, the initial point cloud data is subjected to straight-through filtering to filter the point cloud data of the ground and the rack frame;

[0021] (33) Perform voxel filtering sampling on the point cloud data after the straight-through filtering, construct a point cloud voxel map, calculate the mean of the point cloud in each voxel, and replace all the point clouds in the voxel with the mean;

[0022] (34) After voxel filtering and sampling, due to the volatility and noise interference of the data collected by the camera each time, the point cloud after voxel filtering is processed by Gaussian filtering;

[0023] (35) Performing normal vector filtering on the point cloud data after the Gaussian filtering to determine the circular boundary of the tire assembly.

[0024] As further preferred, step (35) comprises the following steps:

[0025] (351) performing a principal component analysis (PCA) on the point cloud data after the Gaussian filtering process to obtain a normal vector of the point cloud data;

[0026] (352) Since the point cloud coordinate system under the unified coordinate system is the robot base coordinate system, and the z-axis of the base coordinate system coincides with the z-axis of the rack coordinate system, in the normal vector filtering process, the angle between the normal vector of point p and the z-axis is used as the filtering object. When the angle is greater than the threshold θ threshold , remove the point, when the angle is less than the threshold θ threshold , then the point is retained, and after the normal vector filtering process, the circular boundary of the tire assembly is obtained.

[0027] As further preferred, the PCA principal component analysis includes:

[0028] Determine a point p, and use a binary tree to find its n neighboring points or neighboring points within a radius of r. Use these n+1 points to calculate its covariance matrix, and thus calculate its eigenvalues ​​and eigenvectors. The eigenvector corresponding to the minimum eigenvalue is the normal vector of point p.

[0029] As a further preferred embodiment, in step 4, fitting a three-dimensional circle based on the boundary point cloud using a RANSAC algorithm to obtain the spatial posture of the tire assembly includes:

[0030] (41) Perform Euclidean clustering on the point cloud data after normal vector filtering, based on the maximum number of tire assemblies n in each layer. max Segment the individual point clouds of multiple tire assemblies to facilitate subsequent calculation of their poses tire by tire;

[0031] (42) For each tire assembly point cloud data, calculate its boundary point cloud based on the angular clearance algorithm, and extract its tire assembly boundary point cloud one by one;

[0032] (43) After getting n max After obtaining the boundary point cloud of the target workpiece, the RANSAC fitting three-dimensional circle algorithm is used to calculate the center and axis direction of the three-dimensional circle, that is, the spatial position and orientation of the tire assembly.

[0033] As further preferred, in step (42), the angular clearance algorithm includes:

[0034] Traverse the tire assembly point cloud, assuming that the current point is p, and find n neighboring points in its domain or all points in the domain with a radius of r as its neighboring points according to the binary tree. Take the current point as the coordinate origin, map the n domain points to the plane perpendicular to the normal vector of point p, and construct the base coordinate system. At this time, the coordinates of each neighboring point are n=1...n, Sort from largest to smallest, As the angular gap between adjacent points, determine the maximum angular gap When θ max >θ threshold When point p is a boundary point, when θ max ≤θ threshold When , point p is a non-boundary point;

[0035] Preferably, in step (43), the RANSAC three-dimensional circle fitting algorithm process is as follows:

[0036] (431)Defining the mathematical model of a three-dimensional circle requires the spatial coordinates of at least three points;

[0037] (432) In a single cycle, n points are randomly selected from the boundary point cloud, where n ≥ 3, and the three-dimensional circle model and model parameters are calculated using the least squares method;

[0038] (433) Traverse the boundary point cloud, assuming that the current point is p, when the distance between p and the three-dimensional circle model is less than the threshold l threshold , point p is recorded as an inner point, when the distance between p and the three-dimensional circle model is greater than or equal to the threshold l threshold , record point p as an external point, and count the number of internal points;

[0039] (434) Repeat steps (432) and (433), when the number of internal points in the k+1th cycle is greater than the number of internal points in the kth cycle, the optimal three-dimensional circle model parameters are updated to the results of the k+1th cycle;

[0040] (435) A loop termination condition is set. When the number of loops reaches the loop termination condition, after the loop ends, the three-dimensional circle model with the largest number of internal points is taken as the final solution model.

[0041] As further preferred, step five comprises the following steps:

[0042] (51) In the robot base coordinate system, according to the position of the tire assembly rack and the distribution law of the tire assembly, the (x, y) coordinates of the tire assembly are counted, and the average value of the (x, y) coordinates of the tire assembly is used as its position estimate. At the same time, the depth value z of the tire assembly is distributed in a discrete and equidistant interval, so that the above (x, y) coordinates and z coordinates are written into a configuration file to obtain the (x, y, z) coordinates of the n tire assemblies of the i-th single layer;

[0043] (52) According to the i-th layer, the estimated z-axis depth value of its configuration file can be known. Combined with the estimated coordinate value (x, y) in the configuration file, n estimated tire assembly coordinate values ​​(x, y, z) are obtained. The calculated value is traversed to find the shortest distance between it and the estimated value. The sum of the shortest distances of the n points is counted, that is, the sum of the errors between the calculated value and the estimated value is obtained. When the sum of the errors is greater than or equal to the preset threshold, the current calculation has an error, and returns to step three. When the sum of the errors is less than the preset threshold, the current calculation passes.

[0044] According to another aspect of the present invention, there is also provided a spatial posture recognition system for depalletizing a tire assembly industrial robot, comprising:

[0045] The first main control module is used to collect the point cloud data of the rack tire assembly according to the configuration file;

[0046] The second main control module is used to measure the calibration plate image data at n positions of the robot end to calibrate the relationship between the point cloud camera coordinate system and the robot end flange coordinate system n≥3;

[0047] The third main control module is used to After preprocessing the point cloud data, normal vector filtering is performed to determine the circular boundary of the tire assembly;

[0048] The fourth main control module is used to obtain the boundary point cloud of the tire assembly through boundary detection, and fit a three-dimensional circle based on the boundary point cloud using the RANSAC algorithm to obtain the spatial position and posture of the tire assembly;

[0049] The fifth main control module is used to calculate the estimated posture of the tire assembly according to the position of the tire assembly rack and the distribution pattern of the tire assembly, and determine whether the spatial posture is correct according to the error between the estimated posture and the spatial posture. If so, output the result, otherwise trigger the third main control module to re-run the calculation.

[0050] In general, the above technical solution conceived by the present invention has the following technical advantages compared with the prior art:

[0051] 1. The present invention uses point cloud processing technology to solve the problem of posture recognition of tire assemblies. By loading a point cloud camera on the end fixture of the industrial robot to collect point cloud data of the tire assembly from viewpoint to viewpoint, and using point cloud related preprocessing algorithms such as voxel filtering and model fitting algorithms such as normal vector filtering and RANSAC three-dimensional circle fitting, the posture recognition and destacking and loading of the tire assembly are finally realized. This example can be used in automobile assembly line systems to load and schedule the assembled tire assemblies, improve the automation level of related production lines, enhance the robustness of the automobile assembly field, and effectively solve many shortcomings of traditional manual handling such as efficiency and stability. It currently has great industrial application value.

[0052] 2. The present invention integrates advanced visual recognition technology and posture estimation algorithms. This solution can achieve rapid and accurate identification and positioning of tire assemblies, thereby significantly improving the automation level of industrial robots in depalletizing operations. The improvement in the degree of automation not only reduces manual intervention and labor intensity, but also shortens the operation cycle and improves production efficiency.

[0053] 3. The present invention uses advanced technologies such as PCA principal component analysis, RANSAC algorithm and angular clearance algorithm. The solution can accurately extract the boundary of the tire assembly from complex point cloud data and fit its three-dimensional spatial posture. The combined use of these methods improves the accuracy of posture recognition, and due to the robustness of the algorithm, it can maintain a high recognition accuracy even when the point cloud data is noisy and incomplete.

[0054] 4. The present invention reduces the reliance on manual operation through intelligent posture recognition and destacking process, thereby reducing labor costs. At the same time, through precise posture control and optimized destacking strategy, it reduces material waste and equipment loss caused by misidentification or improper operation, further reducing production costs. In addition, the introduction of an automated destacking system also helps to improve operational safety and reduce the occurrence of industrial accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flow chart of a tire assembly posture recognition method according to an embodiment of the present invention;

[0056] Figure 2 is a flow chart of robot hand-eye calibration according to an embodiment of the present invention;

[0057] Figure 3 is a grayscale image of a calibration plate according to an embodiment of the present invention;

[0058] Figure 4 is a binary image of a calibration plate according to an embodiment of the present invention;

[0059] Figure 5 The calibration point detection and drawing result image of the calibration plate according to an embodiment of the present invention;

[0060] Figure 6 is a flowchart of point cloud data preprocessing according to an embodiment of the present invention;

[0061] Figure 7 It is an image in which multiple point clouds are unified into robot base coordinates according to an embodiment of the present invention;

[0062] Figure 8 is a point cloud image after through filtering according to an embodiment of the present invention;

[0063] Fig. 9 is a flow chart of tire assembly posture calculation based on preprocessed point cloud according to an embodiment of the present invention;

[0064] Fig.10 is a preprocessed point cloud normal vector calculation graph according to an embodiment of the present invention;

[0065] Fig.11 is a result image after filtering the point cloud normal vector according to an embodiment of the present invention;

[0066] Fig.12 The point cloud boundary calculation image after filtering the normal vector according to an embodiment of the present invention;

[0067] Fig.13 is a calculation result image of a single-layer tire assembly according to an embodiment of the present invention;

[0068] Fig.14 It is a flow chart for verifying the algorithm calculation results described in one embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0070] Example 1

[0071] like Figure 1 As shown, an embodiment of the present invention provides a spatial posture recognition method for depalletizing a tire assembly industrial robot, comprising:

[0072] Step 1: Collect the point cloud data of the rack tire assembly viewpoint by viewpoint according to the configuration file.

[0073] In this step, a configuration file is used to record multiple viewpoint positions of each layer of tires, and the viewpoints of the current layer are traversed to collect the complete point cloud data of the current layer.

[0074] More specifically, when a forklift transports the tire assembly rack to a designated area, the point cloud camera is mounted on the robot's end fixture and is limited by the field of view of the point cloud camera. It is therefore impossible to obtain the complete point cloud data of the tire assembly on the upper layer of the rack after a single measurement. Therefore, a configuration file (such as txt, json, xml, etc.) is used. The configuration file records multiple viewpoint positions of each layer of tires, and the complete point cloud data of the current layer is collected by traversing the viewpoints of the current layer.

[0075] In one embodiment of the present invention, the ROI of the target workpiece in the depth map is obtained by a convolutional neural network. The depth map is converted into the original point cloud data of the target workpiece according to the depth camera intrinsic parameter matrix and the ROI of the target workpiece in the depth map. More specifically,

[0076] (11) Collecting the depth map data of the tire assembly under actual working conditions, marking the ROI at the top of the depth map where the tire skin can be captured, constructing a tire assembly depth map dataset, and using the dataset to train a convolutional neural network model;

[0077] (12) The depth map of the target workpiece is input into the trained convolutional neural network model to obtain the ROI where the target workpiece is located in the depth map.

[0078] More specifically, in one embodiment, a mask is created based on the ROI where the target workpiece is located in the depth map, and the depth map is filtered based on the mask. During the filtering process, the area outside the ROI is set to 0; the filtered depth map data is converted into point cloud data based on the intrinsic parameter matrix coefficients of the depth camera.

[0079] Optionally, the above also includes:

[0080] (calibrate the internal parameter matrix coefficients of the depth camera, the internal parameter matrix coefficients include: x-axis focal length f x , y-axis focal length f y , x-axis pixel offset c x , y-axis pixel offset c y ;

[0081] Loop through the depth map pixel points to obtain their pixel coordinates (u, v) and the grayscale value at the pixel point, that is, the depth value at the point;

[0082] If the depth value of the point is set to 0, it is skipped. Otherwise, the point cloud coordinates corresponding to the pixel point are calculated, and the original point cloud data of the target workpiece is obtained. The calculation formula of the point cloud coordinates corresponding to the pixel point includes:

[0083]

[0084] Among them, (X, Y, Z) is the point cloud coordinate corresponding to the pixel point.

[0085] More specifically, it also includes:

[0086] The depth map of the tire assembly is captured from different angles by multiple depth cameras, and the depth maps of the tire assembly captured from different angles are fused using a perspective fusion weight calculation formula, wherein the perspective fusion weight calculation formula includes:

[0087]

[0088] Where Wi is the weight of the i-th view, Di is the distance from the i-th view to the target tire, Dj is the distance from the j-th view to the target tire, σ is the standard deviation, which is used to adjust the sensitivity of weight distribution, and n is the total number of view angles;

[0089] The fused depth map is:

[0090]

[0091] Where Dfused(u,v) is the depth value of the fused depth map at the pixel coordinate (u,v), Wi is the weight of the i-th perspective, and Di,depth(u,v) is the depth value of the depth map of the i-th perspective at the pixel coordinate (u,v).

[0092] Step 2: Measure the calibration plate image data at n positions of the robot end to calibrate the relationship between the point cloud camera coordinate system and the robot end flange coordinate system n≥3. Specifically:

[0093] (21) Traverse the n end flange positions of the robot, record the n positions, measure the image data of the calibration plate, and perform grayscale processing;

[0094] (22) Perform Gaussian blur processing and binarization processing on the grayscale image of the calibration plate;

[0095] (23) Calculate the robot hand-eye relationship based on the calibration plate image after binarization processing and the camera internal parameter matrix and distortion coefficient to obtain the relationship between the camera coordinate system and the robot end flange coordinate system

[0096] In this step, the grayscale image of the calibration plate is denoised based on the Gaussian kernel and then binarized to improve the recognition accuracy and efficiency of the calibration plate image, thereby improving the robustness of the subsequent calculation process.

[0097] Secondly, after detecting the image coordinates of the calibration points of the binary image of the calibration plate after preprocessing, its sub-pixel coordinates are calculated, and its external parameter matrix is ​​calculated based on the camera's intrinsic parameter matrix and distortion coefficient, that is, the relationship between the calibration plate coordinate system and the camera coordinate system Finally, the hand-eye relationship between the point cloud camera and the robot end is calculated based on the end flange pose and the calculated camera extrinsic matrix, that is, the relationship between the point cloud camera coordinate system and the robot end flange coordinate system.

[0098] More specifically, the computational relationship Including: Generate 3D coordinates of calibration points based on the inherent geometric parameters of the calibration plate. Detect the image coordinates of the calibration points in the binary image of the calibration plate and calculate their sub-pixel coordinates. Calculate the corresponding camera extrinsic matrix based on the image coordinates of the calibration points, the camera intrinsic matrix and the distortion coefficient. Calculate the robot hand-eye relationship based on n camera extrinsic matrices and n robot end flange positions, that is, the relationship between the point cloud camera coordinate system and the robot end flange coordinate system

[0099] Step 3: Based on the relationship The point cloud data is pre-processed and then subjected to normal vector filtering to determine the circular boundary of the tire assembly. The pre-processing includes: unified coordinate system, straight-through filtering, voxel sampling, Gaussian filtering, normal vector calculation, etc.

[0100] Specifically, in one embodiment of the present invention, the normal vector filtering process after preprocessing the point cloud data includes the following steps:

[0101] (31) Based on the robot hand-eye relationship Unify the multiple point clouds obtained in step 1 into the robot base coordinate system to obtain complete point cloud data;

[0102] (32) Based on the XYZ coordinate range of the rack spatial position, the initial point cloud data is subjected to straight-through filtering to filter the point cloud data of the ground and the rack frame;

[0103] (33) Perform voxel filtering sampling on the point cloud data after the straight-through filtering, construct a point cloud voxel map, calculate the mean of the point cloud in each voxel, and replace all the point clouds in the voxel with the mean;

[0104] (34) After voxel filtering and sampling, due to the volatility and noise interference of the data collected by the camera each time, the point cloud after voxel filtering is processed by Gaussian filtering;

[0105] (35) Performing normal vector filtering on the point cloud data after the Gaussian filtering to determine the circular boundary of the tire assembly.

[0106] Step (35) comprises the following steps:

[0107] (351) performing a principal component analysis (PCA) on the point cloud data after the Gaussian filtering process to obtain a normal vector of the point cloud data;

[0108] (352) Since the point cloud coordinate system under the unified coordinate system is the robot base coordinate system, and the z-axis of the base coordinate system coincides with the z-axis of the rack coordinate system, in the normal vector filtering process, the angle between the normal vector of point p and the z-axis is used as the filtering object. When the angle is greater than the threshold θ threshold , remove the point, when the angle is less than the threshold θ threshold , then the point is retained, and after the normal vector filtering process, the circular boundary of the tire assembly is obtained.

[0109] Optionally, in one embodiment, the PCA principal component analysis includes: determining a point p, searching for its n neighboring points or neighboring points within a region with a radius of r through a binary tree, calculating its covariance matrix with these n+1 points, and thus calculating its eigenvalues ​​and eigenvectors, and the eigenvector corresponding to the minimum eigenvalue is the normal vector of point p.

[0110] Step 4: Obtain the boundary point cloud of the tire assembly through boundary detection, and fit a three-dimensional circle based on the boundary point cloud using the RANSAC algorithm to obtain the spatial pose of the tire assembly.

[0111] Optionally, fitting a three-dimensional circle based on the boundary point cloud using a RANSAC algorithm to obtain the spatial position and posture of the tire assembly includes:

[0112] (41) Perform Euclidean clustering on the point cloud data after normal vector filtering, based on the maximum number of tire assemblies n in each layer. max Segment the individual point clouds of multiple tire assemblies to facilitate subsequent calculation of their poses tire by tire;

[0113] (42) For each tire assembly point cloud data, calculate its boundary point cloud based on the angular clearance algorithm, and extract its tire assembly boundary point cloud one by one;

[0114] (43) After getting n max After obtaining the boundary point cloud of the target workpiece, the RANSAC fitting three-dimensional circle algorithm is used to calculate the center and axis direction of the three-dimensional circle, that is, the spatial position and orientation of the tire assembly.

[0115] In step (42), the angular clearance algorithm includes:

[0116] Traverse the tire assembly point cloud, assuming that the current point is p, and find n neighboring points in its domain or all points in the domain with a radius of r as its neighboring points according to the binary tree. Take the current point as the coordinate origin, map the n domain points to the plane perpendicular to the normal vector of point p, and construct the base coordinate system. At this time, the coordinates of each neighboring point are n=1…n, Sort from largest to smallest, As the angular gap between adjacent points, determine the maximum angular gap When θ max >θ threshold When point p is a boundary point, when θ max ≤θ threshold When , point p is a non-boundary point.

[0117] Preferably, in step (43), the RANSAC three-dimensional circle fitting algorithm process is as follows:

[0118] (431) Defining a mathematical model of a three-dimensional circle requires the spatial coordinates of at least three points;

[0119] (432) In a single cycle, n points are randomly selected from the boundary point cloud, where n ≥ 3, and the three-dimensional circle model and model parameters are calculated using the least squares method;

[0120] (433) Traverse the boundary point cloud, assuming that the current point is p, when the distance between p and the three-dimensional circle model is less than the threshold l threshold , point p is recorded as an inner point, when the distance between p and the three-dimensional circle model is greater than or equal to the threshold l threshold , record point p as an external point, and count the number of internal points;

[0121] (434) Repeat steps (432) and (433), when the number of internal points in the k+1th cycle is greater than the number of internal points in the kth cycle, the optimal three-dimensional circle model parameters are updated to the results of the k+1th cycle;

[0122] (435) A loop termination condition is set. When the number of loops reaches the loop termination condition, after the loop ends, the three-dimensional circle model with the largest number of internal points is taken as the final solution model.

[0123] Step 5: Calculate the estimated posture of the tire assembly according to the position of the tire assembly rack and the distribution law of the tire assembly, and determine whether the spatial posture is correct according to the error between the estimated posture and the spatial posture. If so, output the result, otherwise return to step 3 to re-determine the circular boundary of the tire assembly. The details are as follows:

[0124] (51) In the robot base coordinate system, according to the position of the tire assembly rack and the distribution law of the tire assembly, the (x, y) coordinates of the tire assembly are counted, and the average value of the (x, y) coordinates of the tire assembly is used as its position estimate. At the same time, the depth value z of the tire assembly is distributed in a discrete and equidistant interval, so that the above (x, y) coordinates and z coordinates are written into a configuration file to obtain the (x, y, z) coordinates of the n tire assemblies of the i-th single layer;

[0125] (52) According to the i-th layer, the estimated z-axis depth value of its configuration file can be known. Combined with the estimated coordinate value (x, y) in the configuration file, n estimated tire assembly coordinate values ​​(x, y, z) are obtained. The calculated value is traversed to find the shortest distance between it and the estimated value. The sum of the shortest distances of the n points is counted, that is, the sum of the errors between the calculated value and the estimated value is obtained. When the sum of the errors is greater than or equal to the preset threshold, the current calculation is wrong and returns to step three. When the sum of the errors is less than the preset threshold, the current calculation passes, that is, the spatial posture calculated in step four is correct.

[0126] Example 2

[0127] This embodiment provides a spatial posture recognition system for depalletizing a tire assembly industrial robot, the system being used to implement a method of any one of the above embodiments or a combination of multiple embodiments, including:

[0128] The first main control module is used to collect the point cloud data of the rack tire assembly according to the configuration file;

[0129] The second main control module is used to measure the calibration plate image data at n positions of the robot end to calibrate the relationship between the point cloud camera coordinate system and the robot end flange coordinate system n≥3;

[0130] The third main control module is used to After preprocessing the point cloud data, normal vector filtering is performed to determine the circular boundary of the tire assembly;

[0131] The fourth main control module is used to obtain the boundary point cloud of the tire assembly through boundary detection, and fit a three-dimensional circle based on the boundary point cloud using the RANSAC algorithm to obtain the spatial position and posture of the tire assembly;

[0132] The fifth main control module is used to calculate the estimated posture of the tire assembly according to the position of the tire assembly rack and the distribution pattern of the tire assembly, and determine whether the spatial posture is correct according to the error between the estimated posture and the spatial posture. If so, output the result, otherwise trigger the third main control module to re-run the calculation.

[0133] Example 3

[0134] This embodiment provides a spatial posture recognition method for depalletizing a tire assembly industrial robot. Figure 1 As shown, including:

[0135] Step 1: Collect the point cloud data of the tire assembly on the rack. When the tire assembly rack is moved to the designated area by a forklift, the point cloud camera is installed on the end fixture of the robot. At the same time, due to the limited field of view of the point cloud camera, it is impossible to obtain the complete point cloud data of the upper layer of the rack tire assembly after a single measurement. Therefore, a configuration file (such as txt, json, xml, etc.) is used. The configuration file records multiple viewpoint positions of each layer of tires, and traverses the viewpoints of the current layer to collect the complete point cloud data of the current layer.

[0136] Step 2: After measuring the calibration plate image data at n (n≥3) positions of the robot end and preprocessing it, calibrate the relationship between the point cloud camera coordinate system and the robot end flange coordinate system like Figure 2 shown.

[0137] First, the calibration plate image is preprocessed, specifically including:

[0138] Traverse the robot's n (n ≥ 3) end flange positions and record n ​​positions, measure the calibration plate image data, and perform grayscale processing, such as Figure 3 As shown. Traverse the robot's n (n ≥ 3) postures, read the calibration plate image data, and convert the three-channel data into single-channel grayscale data to meet the subsequent sub-pixel point calculation.

[0139] Secondly, the grayscale image of the calibration plate is Gaussian blurred and binarized, as follows: Figure 4 As shown in the figure, the grayscale image of the calibration plate is denoised based on the Gaussian kernel and then binarized to improve the recognition accuracy and efficiency of the calibration plate image, thereby improving the robustness of the subsequent calculation process.

[0140] Finally, after detecting the image coordinates of the calibration points of the binary image of the calibration plate after preprocessing, its sub-pixel coordinates are calculated, and its external parameter matrix is ​​calculated based on the camera's intrinsic parameter matrix and distortion coefficient, that is, the relationship between the calibration plate coordinate system and the camera coordinate system. Finally, the hand-eye relationship between the point cloud camera and the robot end is calculated based on the end flange pose and the calculated camera extrinsic matrix, that is, the relationship between the point cloud camera coordinate system and the robot end flange coordinate system. Specifically include:

[0141] The 3D coordinates of the calibration points are generated according to the inherent geometric parameters of the calibration plate.

[0142] Detect the image coordinates of the calibration points in the binary image of the calibration plate and calculate their sub-pixel coordinates, such as Figure 5 shown.

[0143] The corresponding camera extrinsic parameter matrix is ​​calculated based on the image coordinates of the calibration points, the camera intrinsic parameter matrix and the distortion coefficient.

[0144] The robot hand-eye relationship is calculated based on the n camera external parameter matrices and the n robot end flange positions, that is, the relationship between the point cloud camera coordinate system and the robot end flange coordinate system.

[0145] Step 3: Perform data preprocessing steps such as unified coordinate system, straight-through filtering, voxel sampling, Gaussian filtering, normal vector calculation, etc. on the point cloud data. Figure 6 shown.

[0146] (31): The multiple point cloud data collected from each viewpoint in step 1 are not in the same coordinate system, but in their respective camera coordinate systems. Therefore, it is necessary to transform the multiple point clouds into the robot base coordinate system according to the robot hand-eye relationship and viewpoint pose calibrated in step 1 to unify the point cloud data coordinate system, such as Figure 7 shown.

[0147] For the complete point cloud data, it includes the point cloud of the main target workpiece (i.e., tire assembly), the background point cloud, and the noise point cloud. The initial point cloud data cannot directly calculate the pose of the target workpiece, so the preprocessing process of the initial point cloud is necessary.

[0148] After obtaining the complete point cloud data in the unified coordinate system, the complete point cloud data needs to be preprocessed to accelerate the pose calculation method and improve the robustness of the pose calculation system. The point cloud data preprocessing process specifically includes:

[0149] (32): Based on the XYZ coordinate range of the rack spatial position, the initial point cloud data is subjected to straight-through filtering to filter the point cloud data of the ground and the rack frame, paving the way for subsequent Euclidean clustering, such as Figure 8 shown.

[0150] (33): Perform voxel filtering sampling on the point cloud after direct filtering, construct a point cloud voxel map, calculate the mean of the point cloud in each voxel, and replace all the point clouds in the voxel with the mean. In this way, data redundancy is reduced while maintaining most of the geometric information of the original data, so as to speed up the subsequent calculation process.

[0151] (34): After voxel filtering sampling, due to the volatility and noise interference of the camera data collected each time, the point cloud after voxel filtering is processed by Gaussian filtering to reduce the volatility and noise interference of the point cloud data collected each time.

[0152] After point cloud data preprocessing, it is observed that the point cloud data of the target workpiece (i.e., tire assembly) has no obvious geometric model to match it. At the same time, it can be observed that the edge normal of the target workpiece is continuously distributed. Therefore, a normal vector filter function is designed to filter and obtain the circular boundary of the target workpiece, which can be used as the basis for subsequent pose calculation. The pose calculation process is as follows: Fig. 9 shown.

[0153] As shown in Figure x, normal vector filtering is performed on the preprocessed point cloud, specifically including:

[0154] The normal vector calculation method based on PCA principal component analysis is used to calculate the normal vector of the preprocessed point cloud point by point, such as Fig.10 shown.

[0155] PCA principal component analysis determines a point p, finds its n neighboring points or neighboring points within a range with a radius of r through a binary tree, and calculates its covariance matrix with these n+1 points, thereby calculating its eigenvalues ​​and eigenvectors. The eigenvector corresponding to the minimum eigenvalue is the normal vector of point p.

[0156] Since the point cloud coordinate system under the unified coordinate system is the robot base coordinate system, and the z-axis of the base coordinate system coincides with the z-axis of the material rack (i.e., workpiece) coordinate system, in the normal vector filtering process, the angle between the normal vector of point p and the z-axis is used as the filtering object. When the angle is greater than the threshold θ threshold , remove the point, when the angle is less than the threshold θ threshold , then keep this point, and the filtering result is as follows Fig.11 In this embodiment, the threshold value θ threshold Set to between 20-40°.

[0157] After the normal vector filtering process, the target workpiece point cloud obtained has a clearer circular boundary. After obtaining its boundary point cloud through boundary detection, the center coordinates and axis direction of its boundary can be obtained according to the RANSAC three-dimensional circle fitting method, that is, the position and posture result of the target workpiece (tire assembly).

[0158] The method for calculating the pose of the point cloud target workpiece after normal vector filtering specifically includes:

[0159] The point cloud data after normal vector filtering is clustered in Euclidean order according to the maximum number of tire assemblies in each layer n. max Segment the individual point clouds of multiple target workpieces to facilitate subsequent calculation of their poses one by one.

[0160] In the loop, for each target workpiece point cloud data, its boundary point cloud is calculated based on the angular clearance algorithm, and the target workpiece boundary point cloud is extracted one by one, such as Fig.12 shown.

[0161] The angular clearance algorithm is to traverse the point cloud, assuming that the current point is p, and find n neighboring points in its area or all points in the area with a radius of r as its neighboring points according to the binary tree. With the current point as the coordinate origin, the n area points are mapped to the plane perpendicular to the normal vector of point p, and the base coordinate system is constructed. At this time, the coordinates of each neighboring point are n=1…n, Sort from largest to smallest, As the angular gap between adjacent points, determine the maximum angular gap When θ max >θ threshold When point p is a boundary point, when θ max ≤θ threshold When , point p is a non-boundary point.

[0162] In getting n max After obtaining the boundary point cloud of the target workpiece, the RANSAC fitting three-dimensional circle algorithm is used to calculate the center and axis direction of the three-dimensional circle, that is, the position and posture result of the target workpiece (tire assembly), such as Fig.13 shown.

[0163] The RANSAC three-dimensional circle fitting algorithm process is as follows:

[0164] Defining a mathematical model of a three-dimensional circle requires the spatial coordinates of at least three points.

[0165] In a single cycle, n (n≥3) points are randomly selected from the boundary point cloud, and the three-dimensional circle mathematical model is calculated using the least squares method.

[0166] Traverse the boundary point cloud, assuming that the current point is p, when the distance between p and the 3D circle model is less than the threshold l threshold , point p is recorded as an inner point, when the distance between p and the three-dimensional circle model is greater than or equal to the threshold l threshold , denote point p as an external point, and count the number of internal points.

[0167] Traversing the above loop, when the number of internal points in the k+1th loop is greater than the number of internal points in the kth loop, the optimal three-dimensional circle model parameters are updated to the k+1th loop result. The loop termination condition is that the number of loops reaches the preset value.

[0168] After the loop is completed, the three-dimensional circle model with the largest number of internal points is taken as the final solution model.

[0169] Step 4: After calculating multiple poses of a single-layer target workpiece (i.e., tire assembly), a method for judging the correctness of the calculation results is still required. By detecting the correctness of the calculation of the aforementioned algorithm, the problem of interference and collision between the fixture and the environment or the target workpiece can be avoided, thereby enhancing the stability of the pose recognition algorithm. Fig.14 shown.

[0170] Because the tire assembly rack is transported to the designated area by a forklift each time, and because the tire assembly is arranged in a certain regularity, there is a potential estimated value for the position of the tire assembly. Therefore, it is only necessary to compare the error between the calculated value and the estimated value of the posture detection algorithm to determine the correctness of the calculation result.

[0171] The specific process is as follows:

[0172] (41): In the robot base coordinate system, based on the position of the tire assembly rack and the approximate distribution pattern of the tire assembly, its (x, y) coordinates are counted and its average value is used as the estimated position value. At the same time, the depth value z of the tire assembly is distributed in discrete equidistant intervals, so that the above (x, y) coordinates and z coordinates can be written into a configuration file.

[0173] (42): After calculation by the aforementioned detection algorithm, the (x, y, z) coordinates of the n tire assemblies in the i-th single layer are obtained. Based on the i-th layer, the estimated z-axis depth value of the configuration file can be known. Combined with the estimated coordinate value (x, y) in the configuration file, n estimated tire assembly coordinate values ​​(x, y, z) can also be obtained. The calculated value is traversed to find the shortest distance between it and the estimated value. The sum of the shortest distances of the n points is calculated to obtain the sum of the errors between the calculated value and the estimated value. When the sum of the errors is greater than or equal to the preset threshold, the current calculation is wrong. When the sum of the errors is less than the preset threshold, the current calculation passes.

[0174] So far, this example uses point cloud processing technology to solve the posture recognition problem of tire assemblies. By loading a point cloud camera on the end fixture of the industrial robot to collect the point cloud data of the tire assembly from viewpoint to viewpoint, and using point cloud related preprocessing algorithms such as voxel filtering and model fitting algorithms such as normal vector filtering and RANSAC three-dimensional circle fitting, the posture recognition and depalletizing and loading of the tire assembly are finally realized. This example can be used in automobile assembly line systems to load and schedule the assembled tire assemblies, improve the automation level of related production lines, enhance the robustness of the automobile assembly field, and effectively solve many shortcomings of traditional manual handling such as efficiency and stability. It has great industrial application value at present.

[0175] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A spatial posture recognition method for depalletizing tire assembly industrial robots, characterized in that: The following steps are involved: Step 1: Collect the point cloud data of the tire assembly of the rack viewpoint by viewpoint according to the configuration file; Step 2: Measure the calibration plate image data at n positions of the robot end to calibrate the relationship between the point cloud camera coordinate system and the robot end flange coordinate system n≥3; Step 3: Based on the relationship After preprocessing the point cloud data, normal vector filtering is performed to determine the circular boundary of the tire assembly; Step 4: Obtain the boundary point cloud of the tire assembly through boundary detection, and fit a three-dimensional circle based on the boundary point cloud using the RANSAC algorithm to obtain the spatial pose of the tire assembly; Step five, according to the position of the tire assembly rack and the distribution pattern of the tire assembly, calculate the estimated posture of the tire assembly, and determine whether the spatial posture is correct according to the error between the estimated posture and the spatial posture. If so, output the result, otherwise return to step three.

2. A spatial posture recognition method for depalletizing tire assembly industrial robots according to claim 1, characterized in that: Step one also includes the following steps: A configuration file is used to record multiple viewpoint positions of each tire layer, and the viewpoints of the current layer are traversed to collect the complete point cloud data of the current layer.

3. The spatial posture recognition method for depalletizing tire assembly industrial robots according to claim 1, characterized in that: Step 2 includes the following steps: (21) Traverse the n end flange positions of the robot, record the n positions, measure the image data of the calibration plate, and perform grayscale processing; (22) Perform Gaussian blur processing and binarization processing on the grayscale image of the calibration plate; (23) Calculate the robot hand-eye relationship based on the calibration plate image after binarization processing and the camera internal parameter matrix and distortion coefficient to obtain the relationship between the camera coordinate system and the robot end flange coordinate system 4. The spatial posture recognition method for depalletizing tire assembly industrial robots according to claim 1, characterized in that: In step three, the normal vector filtering process after preprocessing the point cloud data includes the following steps: (31) Based on the robot hand-eye relationship Unify the multiple point clouds obtained in step 1 into the robot base coordinate system to obtain complete point cloud data; (32) Based on the XYZ coordinate range of the rack spatial position, the initial point cloud data is subjected to straight-through filtering to filter the point cloud data of the ground and the rack frame; (33) Perform voxel filtering sampling on the point cloud data after the straight-through filtering, construct a point cloud voxel map, calculate the mean of the point cloud in each voxel, and replace all the point clouds in the voxel with the mean; (34) After voxel filtering and sampling, due to the volatility and noise interference of the data collected by the camera each time, the point cloud after voxel filtering is processed by Gaussian filtering; (35) Performing normal vector filtering on the point cloud data after the Gaussian filtering to determine the circular boundary of the tire assembly.

5. The spatial posture recognition method for depalletizing tire assembly industrial robots according to claim 4, characterized in that: Step (35) comprises the following steps: (351) performing a principal component analysis (PCA) on the point cloud data after the Gaussian filtering process to obtain a normal vector of the point cloud data; (352) Since the point cloud coordinate system under the unified coordinate system is the robot base coordinate system, and the z-axis of the base coordinate system coincides with the z-axis of the rack coordinate system, in the normal vector filtering process, the angle between the normal vector of point p and the z-axis is used as the filtering object. When the angle is greater than the threshold θ threshold , remove the point, when the angle is less than the threshold θ threshold , then the point is retained, and after the normal vector filtering process, the circular boundary of the tire assembly is obtained.

6. The spatial posture recognition method for depalletizing tire assembly industrial robots according to claim 6, characterized in that: The PCA principal component analysis includes: Determine a point p, and use a binary tree to find its n neighboring points or neighboring points within a radius of r. Use these n+1 points to calculate its covariance matrix, and thus calculate its eigenvalues ​​and eigenvectors. The eigenvector corresponding to the minimum eigenvalue is the normal vector of point p.

7. The spatial posture recognition method for depalletizing tire assembly industrial robots according to claim 1, characterized in that: In step 4, fitting a three-dimensional circle based on the boundary point cloud using the RANSAC algorithm to obtain the spatial position and posture of the tire assembly includes: (41) Perform Euclidean clustering on the point cloud data after normal vector filtering, based on the maximum number of tire assemblies n in each layer. max Segment the individual point clouds of multiple tire assemblies to facilitate subsequent calculation of their poses tire by tire; (42) For each tire assembly point cloud data, calculate its boundary point cloud based on the angular clearance algorithm, and extract its tire assembly boundary point cloud one by one; (43) After getting n max After obtaining the boundary point cloud of the target workpiece, the RANSAC fitting three-dimensional circle algorithm is used to calculate the center and axis direction of the three-dimensional circle, that is, the spatial position and orientation of the tire assembly.

8. The spatial posture recognition method for depalletizing tire assembly industrial robots according to claim 7, characterized in that: In step (42), the angular clearance algorithm includes: Traverse the tire assembly point cloud, assuming that the current point is p, and find n neighboring points in its domain or all points in the domain with a radius of r as its neighboring points according to the binary tree. Take the current point as the coordinate origin, map the n domain points to the plane perpendicular to the normal vector of point p, and construct the base coordinate system. At this time, the coordinates of each neighboring point are n=1…n, Sort from largest to smallest, As the angular gap between adjacent points, determine the maximum angular gap When θ max >θ threshold When θ max ≤θ threshold When , point p is a non-boundary point; Preferably, in step (43), the RANSAC three-dimensional circle fitting algorithm process is as follows: (431)Defining the mathematical model of a three-dimensional circle requires the spatial coordinates of at least three points; (432) In a single cycle, n points are randomly selected from the boundary point cloud, where n ≥ 3, and the three-dimensional circle model and model parameters are calculated using the least squares method; (433) Traverse the boundary point cloud, assuming that the current point is p, when the distance between p and the three-dimensional circle model is less than the threshold l threshold , point p is recorded as an inner point, when the distance between p and the three-dimensional circle model is greater than or equal to the threshold l threshold , record point p as an external point, and count the number of internal points; (434) Repeat steps (432) and (433), when the number of internal points in the k+1th cycle is greater than the number of internal points in the kth cycle, the optimal three-dimensional circle model parameters are updated to the results of the k+1th cycle; (435) A loop termination condition is set. When the number of loops reaches the loop termination condition, after the loop ends, the three-dimensional circle model with the largest number of internal points is taken as the final solution model.

9. The spatial posture recognition method for depalletizing tire assembly industrial robots according to claim 1, characterized in that: Step five includes the following steps: (51) In the robot base coordinate system, according to the position of the tire assembly rack and the distribution law of the tire assembly, the (x, y) coordinates of the tire assembly are counted, and the average value of the (x, y) coordinates of the tire assembly is used as its position estimate. At the same time, the depth value z of the tire assembly is distributed in a discrete and equidistant interval, so that the above (x, y) coordinates and z coordinates are written into a configuration file to obtain the (x, y, z) coordinates of the n tire assemblies of the i-th single layer; (52) According to the i-th layer, the estimated z-axis depth value of its configuration file can be known. Combined with the estimated coordinate value (x, y) in the configuration file, n estimated tire assembly coordinate values ​​(x, y, z) are obtained. The calculated value is traversed to find the shortest distance between it and the estimated value. The sum of the shortest distances of the n points is counted, that is, the sum of the errors between the calculated value and the estimated value is obtained. When the sum of the errors is greater than or equal to the preset threshold, the current calculation has an error, and returns to step three. When the sum of the errors is less than the preset threshold, the current calculation passes.

10. A spatial posture recognition system for depalletizing tire assembly industrial robots, characterized in that: include: The first main control module is used to collect the point cloud data of the rack tire assembly according to the configuration file; The second main control module is used to measure the calibration plate image data at n positions of the robot end to calibrate the relationship between the point cloud camera coordinate system and the robot end flange coordinate system n≥3; The third main control module is used to After preprocessing the point cloud data, normal vector filtering is performed to determine the circular boundary of the tire assembly; The fourth main control module is used to obtain the boundary point cloud of the tire assembly through boundary detection, and fit a three-dimensional circle based on the boundary point cloud using the RANSAC algorithm to obtain the spatial position and posture of the tire assembly; The fifth main control module is used to calculate the estimated posture of the tire assembly according to the position of the tire assembly rack and the distribution pattern of the tire assembly, and determine whether the spatial posture is correct according to the error between the estimated posture and the spatial posture. If so, output the result, otherwise trigger the third main control module to re-run the calculation.

Images