A 3D object surface defect detection method based on multi-robot arm cooperation

Abstract

The present invention provides a three-dimensional object surface defect detection method based on multi-robot arm cooperation, belonging to the technical field of defect detection. The present invention generates a series of detection points according to the CAD model of the object to be detected, which serve as the position point coordinates for the robot arms to perform defect detection; then, according to the distribution of the detection points, the number of robot arms participating in the task is determined, and the detection points are grouped and assigned to each robot arm; after the assignment is completed, a reasonable robot arm path planning is carried out using a continuously optimized sequential greedy search algorithm, so that the robot arms can achieve full coverage defect detection of the object surface; finally, the task is started, and according to the image information collected by the robot arms, a multi-level defect detection method based on physical optical compensation and dynamic region growth is used for detection, and a visual result report is generated, realizing efficient and accurate detection of three-dimensional object surface defects through multi-robot arm cooperation, and providing a reliable quality detection means for industrial production.

Description

Technical Field

[0001] The present invention belongs to the technical field of defect detection, and particularly relates to a three-dimensional object surface defect detection method based on multi-robot arm cooperation. Background Art

[0002] Defect detection on the surface of an object is one of the key links to ensure product quality. Traditional surface defect detection methods mainly rely on manual visual inspection or single robot arm detection equipment. Manual visual inspection has many limitations, such as low detection efficiency, susceptibility to subjective factors resulting in low accuracy, and being prone to fatigue during long-term work, making it difficult to meet the requirements of large-scale production; while single robot arm detection equipment, although improving the detection efficiency to a certain extent, due to the limitations of its movement range and detection perspective, there are many deficiencies in defect detection on the surface of complex three-dimensional objects, and it is difficult to comprehensively and accurately identify defects in various parts of the object.

[0003] With the continuous development of industrial automation and intelligence, multi-robot arm cooperation systems have gradually been applied to various complex tasks, showing great potential. Multi-robot arm cooperation can give full play to the advantages of each robot arm and, through reasonable task allocation and collaborative control, achieve efficient processing of complex environments and tasks. In the field of three-dimensional object surface defect detection, introducing the concept of multi-robot arm cooperation is expected to break through the bottleneck of traditional detection methods and achieve high-precision and high-efficiency defect detection. In the prior art, although some studies have involved the application of multi-robot arm cooperation in object recognition tasks, most of them focus on tasks such as object grasping and handling, and there is no application of multi-robot arm cooperation in three-dimensional object surface defect detection tasks.

[0004] Therefore, it is necessary to provide a three-dimensional object surface defect detection method based on multi-robot arm cooperation to solve the above problems. Summary of the Invention

[0005] The present invention provides a three-dimensional object surface defect detection method based on multi-robot arm cooperation, which realizes efficient and accurate detection of three-dimensional object surface defects by optimizing the cooperative control strategy, detection path planning, and data processing and analysis process of multi-robot arms, provides a reliable quality detection means for industrial production, and can effectively solve at least one technical problem involved in the background art.

[0006] To solve the above technical problems, the present invention is implemented as follows:

[0007] A three-dimensional object surface defect detection method based on multi-robot arm cooperation includes the following steps:

[0008] Step S1: Use the manifold Gaussian method to map the surface information field of the object to be detected, as well as its mean function and covariance function, into the manifold space. Then, use the Poisson disk sampling algorithm to sample in the manifold space to generate a detection point set.

[0009] Step S2: Determine the workspace of each robotic arm according to its structural parameters. Select the number of robotic arms required to perform the task based on the volume coverage rate of the combined workspace of all robotic arms and the object to be detected. On the basis of the traditional K-means clustering, introduce the curvature distribution entropy to form an improved clustering algorithm. Use the improved clustering algorithm to classify the detection point set into multiple subsets of detection points in different clusters, and assign a subset of detection points to each robotic arm.

[0010] Step S3: Use the continuous optimization sequential greedy search path planning algorithm to plan the optimal path for each robotic arm to scan all the detection points in the corresponding subset of detection points.

[0011] Step S4: Each robotic arm runs according to the optimal path, collects images of the object to be detected at the detection points, and performs surface detection on the object to be detected using a multi-level defect detection method based on physical optical compensation and dynamic region growing.

[0012] Step S5: Generate a detection report based on the detection results of each robotic arm, and visually annotate the defect areas.

[0013] As a preferred improvement, Step S1 specifically includes the following steps:

[0014] Step S11: Establish a CAD model of the object to be detected. Use the manifold Gaussian method to map the surface information field of the CAD model, as well as its mean function and covariance function, into the manifold space. The kernel function used in the manifold space is the geodesic Matern 3 / 2 kernel function, which is expressed as:

[0015]

[0016] In the formula, represents the geodesic distance between any two points in the manifold space; represents the kernel function, which is used to characterize the spatial correlation between any two points in the manifold space; represents the variance of the surface information field; represents the length scale parameter;

[0017] Step S12: Define a dynamic sampling density function. Use the Poisson disk sampling algorithm based on the rejection radius to sample in the manifold space. Adjust the rejection radius of the sampling points according to the dynamic sampling density function, and generate a detection point set that meets the density constraint in the space. The dynamic sampling density function is expressed as:

[0018]

[0019] In the formula, represents the sampling density; represents the basic sampling density; represents the curvature sensitivity coefficient; and respectively represent the principal curvatures of the triangular facets in the manifold space, which are obtained through the surface differential geometry analysis method.

[0020] As a preferred improvement, the working space of the robotic arm is determined as follows: The homogeneous transformation matrices of each joint in the robotic arm are multiplied continuously to obtain the pose matrix of the end effector relative to the base coordinate system; then the angular ranges of each joint are uniformly discretized, and the Monte Carlo random sampling method is used to generate a point cloud dataset of the reachable positions of the end of the robotic arm. The convex hull calculation is performed on the point cloud dataset, and the geometric boundary of the working space of the robotic arm is extracted to obtain a polyhedron model representing the working space of the robotic arm. The polyhedron model consists of a vertex set and a triangular facet set.

[0021] As a preferred improvement, the number of robotic arms required to perform the task is determined as follows:

[0022] (i) Quantify the size of the object to be detected through the bounding box, and calculate the minimum number of robotic arms from the coverage requirements of the object to be detected in the horizontal direction , and the calculation process is expressed as:

[0023]

[0024] In the formula, represents rounding up; and respectively represent the length and width of the bounding box; represents the effective detection range of the robotic arm;

[0025] (ii) Set the upper limit of the number of robotic arms to 2 , and construct a candidate set of the number of robotic arms , where , and calculate the volume coverage rate of the combined working space of the robotic arms with this number and the object to be detected for each candidate solution in the candidate set. The calculation process is expressed as:

[0026]

[0027] In the formula, represents the volume coverage rate of the combined working space of the robotic arms with this number and the object to be detected; ; represents the intersection volume of the combined working space of the robotic arms with this number and the object to be detected; Represents the total volume of the object to be detected;

[0028] (iii) Set the volume coverage threshold, and select the candidate solution with the smallest value from the candidate set on the condition that the volume coverage of the candidate solution is not less than the volume coverage threshold as the number of robotic arms to perform the task. in the candidate set as the number of robotic arms to perform the task.

[0029] As a preferred improvement, the optimization objective of the improved clustering algorithm is expressed as:

[0030]

[0031] In the formula, is the preset number of clusters, which is equal to the number of robotic arms; represents the subset of detection points in the th cluster; represents the coordinates of the center point of the th cluster; represents the th detection point in the detection point set; represents the weight coefficient; represents the curvature distribution entropy, and its calculation method is:

[0032]

[0033] In the formula, is the probability that the curvature of the detection points in the cluster is in the th interval; is the number of intervals of the curvature histogram;

[0034] The optimization process of the improved clustering algorithm is specifically as follows: First, select initial clustering centers, and ensure that each clustering center is located in the common area of the workspaces of the robotic arms; then perform iterative optimization. In the detection point assignment stage, assign each detection point to the cluster with the smallest objective function that is reachable; in the update stage, recalculate the clustering centers and constrain them to be within the workspaces of the corresponding robotic arms; finally, perform post-processing, merge the clusters with too few points and split the clusters with too many points.

[0035] As a preferred improvement, step S3 specifically includes the following steps:

[0036] Step S31, use the sequential greedy search algorithm to sort the detection points required for each robotic arm to obtain the initial path;

[0037] The generation process of the initial path specifically includes the following steps: Given the current detection point , select the next best detection point from the detection point subset , make the robotic wall at the detection point be able to cover the most undetected areas and have the highest information collection efficiency;

[0038] The new information collection of the robotic arm at any detection point position is defined as follows:

[0039]

[0040] In the formula, represents the trace of the matrix; and represent the covariance of the object surface map after and before scanning at the detection point;

[0041] Step S32, taking the path points on the initial path as decision variables, and using a continuous optimization algorithm to optimize the initial path to obtain the optimal solution;

[0042] The generation process of the optimal path specifically includes the following steps: Connect the detection points on the initial trajectory using a polynomial trajectory, obtain the surface area measured along the polynomial trajectory, and optimize the polynomial trajectory with the maximization of the time-averaged area gain U as the optimization objective to generate the optimal path. The time-averaged area gain U is expressed as:

[0043]

[0044] In the formula, represents the surface area measured along the polynomial trajectory.

[0045] As a preferred improvement, in step S3, the planned trajectory needs to add the following constraints:

[0046] The next detection point must be within the line of sight of the current detection point. Among them, the line of sight is judged by sampling positions from the line segment connecting the two detection points and checking whether any sampling position collides with the grid of the object to be detected according to its Euclidean signed distance value. If there is a sampling position that collides with the grid of the object to be detected, it is determined that the next detection point is not within the line of sight of the current detection point; otherwise, it is determined that the next detection point is within the line of sight of the current detection point.

[0047] As a preferred improvement, step S4 specifically includes the following steps:

[0048] Step S41: Convert the vertices of the CAD model of the object to be detected from the base coordinate system to the camera coordinate system. After removing the invisible triangular patches, map the three-dimensional space points to the two-dimensional image plane using the pinhole camera principle, and then rasterize the visible triangular patches using the scan line algorithm to generate a reference image of the object to be detected;

[0049] Step S42: Establish a light compensation model to eliminate the difference between the actual lighting conditions and the theoretical lighting model, and perform light compensation on the collected image to obtain a corrected image. The light compensation model is expressed as:

[0050]

[0051]

[0052] In the formula, 、 represent the image pixel coordinates; represents the light compensation coefficient; represents the gray value of the actual image; represents the expected gray value of the image under the theoretical lighting model; represents a minimum value, used to prevent division by zero errors; represents the diffuse reflection coefficient of the material of the object to be detected, and the value range is ; represents the surface normal vector; represents the normalized light source direction vector, obtained through mirror sphere calibration; represents the specular reflection coefficient; represents the reflected light direction vector; represents the line of sight direction vector; represents the specular highlight exponent, used to control the sharpness of specular reflection;

[0053] Perform Gaussian filtering on to eliminate noise and high-frequency fluctuations, and obtain :

[0054]

[0055] In the formula, represents the corrected light compensation coefficient; represents the Gaussian kernel, used to retain the low-frequency non-uniformity of light; represents the convolution operation;

[0056] The corrected image is expressed as:

[0057]

[0058] Step S43: Calculate the cross-power spectrum of the reference image and the corrected image using the phase correlation method, and obtain the cross-correlation matrix of the reference image and the corrected image through inverse Fourier transform, which is expressed as:

[0059]

[0060] In the formula, represents the cross-correlation matrix of the reference image and the corrected image; and represent the spatial frequency coordinate quantities in the frequency domain, represents the corrected image; represents the reference image; represents the two-dimensional Fourier transform, which converts the image to the frequency domain; represents the complex conjugate operation, which is used to calculate the frequency domain correlation; represents the inverse Fourier transform;

[0061] Step S44: Eliminate the positioning error of the robotic arm and adaptively distinguish noise from real defects through sub-pixel image registration and dynamic threshold, which specifically includes the following process:

[0062] Find the maximum peak coordinate from the cross-correlation matrix , perform quadratic fitting on the peak neighborhood, and obtain the sub-pixel offset ;

[0063] Perform sub-pixel translation correction on the corrected image using bilinear interpolation to obtain the corrected image :

[0064]

[0065] Step S45: Extract the defect area from the actual image collected by the robotic arm, and use morphological processing to eliminate noise and connect real defect areas, which specifically includes the following steps:

[0066] Calculate the difference image between the corrected image and the reference image :

[0067]

[0068] Statistical gray histogram of the difference image , take the average value and standard deviation of the gray histogram, set the threshold to , and pixels exceeding are regarded as defects;

[0069] Perform a closing operation on the difference image, dilate and then erode it using a 5x5 circular kernel structuring element to fill small holes; then perform an opening operation, erode and then dilate it using a 5x5 circular kernel structuring element to eliminate isolated noise points. Then apply the region growing algorithm to accurately extract the continuous defect region. In the morphologically processed image, select pixels with a gray level difference exceeding twice the threshold as the growth starting point, and then check the neighboring pixels. If the gray level difference is less than a certain value, merge them into the growth region; the stopping condition is that the growth reaches the image boundary or 10 consecutive pixels no longer meet the merging condition.

[0070] As a preferred improvement, after step S45, the following steps are further included:

[0071] Step S46, perform quantitative analysis and classification of the defects through geometric parameter calculation and type discrimination rules. The geometric parameter calculation includes the calculation of the defect depth and the defect area, where:

[0072] Defect depth The calculation process is expressed as:

[0073]

[0074] In the formula, represents the number of pixels in the defect region; , respectively represent the gray level values of the reference image and the actually acquired image after alignment at the pixel point ; is the depth conversion coefficient, obtained through calibration;

[0075] Defect area The calculation process is expressed as:

[0076]

[0077] In the formula, represents the image resolution, determined by the camera focal length and the working distance;

[0078] The type of the defect is classified through the discrimination rule:

[0079] Scratch determination is based on the relationship between the major axis length and the minor axis length of the defect region; specifically: if and the depth gradient direction is consistent with the major axis direction, it is determined as a scratch, is a ratio threshold set manually;

[0080] Pit determination is made through the defect area and the depth gradient; specifically: if If the depth gradient is radially distributed along the edge, it is determined as a pit. is an artificially set area threshold.

[0081] Crack determination is based on the actual perimeter of the defect area and the perimeter of the equivalent circle The ratio of which is determined by the number of crack branches; specifically: if and the number of branches is greater than or equal to 3, it is determined as a crack. is an artificially set ratio threshold.

[0082] As a preferred improvement, step S5 specifically includes the following steps:

[0083] Step S51, perform coordinate unification processing on the defect data uploaded by each robotic arm, convert the local detection coordinates to the global coordinate system, and generate a global defect point cloud set;

[0084] The conversion process of local detection coordinates to the global coordinate system is expressed as:

[0085]

[0086] In the formula, represents the coordinates of the defect point in the global coordinate system; represents the coordinates of the defect point in the local detection coordinate system; represents the real-time pose matrix of the end effector, calculated from the structural parameters of the robotic arm; represents the pose matrix of the robotic arm base coordinate system relative to the global coordinate system;

[0087] The global defect point cloud set is expressed as: , , in the formula, represents the total number of local detection points;

[0088] Step S52, determine the triangular patch to which each defect point in the global defect point cloud set belongs, and then perform RGB color coding on the triangular patch to which the defect point belongs according to the defect type to achieve defect annotation;

[0089] The mapping relationship between the defect point and the triangular patch is expressed as:

[0090]

[0091] In the formula, represents the triangular patch to which the defect point belongs; represents the operation of projecting the defect point onto the triangular patch plane; is the triangular patch set of the CAD model of the object to be detected; represents the k th triangular patch;

[0092] The RGB color coding rule is expressed as: the color coding of cracks is (255, 0, 0), the color coding of pits is (0, 0, 255), and the color coding of scratches is (0, 255, 0).

[0093] The beneficial effects of the present invention are as follows:

[0094] (1) Through the dynamic task allocation and robotic arm collaborative scheduling mechanism, multiple robotic arms can participate in the defect detection task simultaneously. Compared with the traditional single robotic arm solution, the task execution efficiency can be significantly improved, and the detection blind area is greatly reduced, improving the coverage rate;

[0095] (2) The present invention adopts a continuously optimized sequential greedy search path planning algorithm, combined with the mean and covariance information of the surface of the object to be detected, reducing the repeated coverage rate of the robotic arm's running path. At the same time, an Euclidean signed distance field is constructed to dynamically adjust the safety distance threshold to achieve a collision-free trajectory and reduce the collision risk;

[0096] (3) The present invention adopts a multi-level detection method based on physical optical compensation and dynamic region growing. By light modeling, the reflection interference on the object surface is eliminated, high-precision sub-pixel registration is achieved, reducing the false detection rate. At the same time, combined with morphological processing, noise interference is effectively suppressed, improving the robustness of defect detection on the surface of complex curved surfaces. Specific Embodiments

[0097] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0098] This embodiment provides a three-dimensional object surface defect detection method based on multi-robotic arm cooperation, including the following steps:

[0099] Step S1: Use the manifold Gaussian method to map the surface information field of the object to be detected, its mean function, and covariance function to the manifold space, and use the Poisson disk sampling algorithm to sample in the manifold space to generate a detection point set.

[0100] Step S1 specifically includes the following steps:

[0101] Step S11: Establish a CAD model of the object to be detected, use the manifold Gaussian method to map the surface information field of the CAD model, its mean function, and covariance function to the manifold space. The kernel function adopted by the manifold space is the geodesic Matern 3 / 2 kernel function, expressed as:

[0102]

[0103] wherein represents the geodesic distance between any two points on the manifold space; represents the kernel function, which is used to characterize the spatial correlation between any two points on the manifold space; represents the variance of the surface information field; represents the length scale parameter;

[0104] Step S12: Define the dynamic sampling density function, and use the Poisson disk sampling algorithm based on the rejection radius to sample in the manifold space. Adjust the rejection radius of the sampling points according to the dynamic sampling density function, and generate a set of detection points that meet the density constraint in the space. The dynamic sampling density function is expressed as:

[0105]

[0106] wherein represents the sampling density; represents the basic sampling density; represents the curvature sensitivity coefficient; and respectively represent the principal curvatures of the triangular facets in the manifold space, which are obtained by the surface differential geometry analysis method.

[0107] The setting of the dynamic sampling density function can ensure a higher sampling density in the key area (high curvature area), and at the same time avoid uneven or repeated distribution of detection points.

[0108] Step S2: Determine the working space of each robotic arm according to the structural parameters of the robotic arm. Select the number of robotic arms required to perform the task based on the volume coverage rate of the combined working space of all robotic arms and the object to be detected. Introduce the curvature distribution entropy on the basis of the traditional K-means clustering to form an improved clustering algorithm, and use the improved clustering algorithm to classify the set of detection points into subsets of detection points in multiple different clusters, and assign a subset of detection points to each robotic arm.

[0109] The working space of the robotic arm is determined as follows: Multiply the homogeneous transformation matrices of the joints in the robotic arm to obtain the pose matrix of the end effector relative to the base coordinate system; Then uniformly discretize the angular ranges of the joints, and use the Monte Carlo random sampling method to generate a point cloud data set of the reachable positions of the end of the robotic arm. Perform a convex hull calculation on the point cloud data set Consisting of a vertex set and a triangular facet set .

[0110] The Monte Carlo random sampling method is a sampling method based on probability statistics and random simulation. In the analysis of the manipulator workspace, the Monte Carlo method randomly samples the angle combinations of each joint of the manipulator, calculates the position of the end effector, and finally determines the reachable workspace range through the point cloud distribution.

[0111] The number of manipulators required to perform the task is determined as follows:

[0112] (i) Quantify the size of the object to be detected through a bounding box, and calculate the minimum number of manipulators from the coverage requirements of the object to be detected in the horizontal direction. The calculation process is expressed as:

[0113]

[0114] In the formula, represents rounding up; , respectively represent the length and width of the bounding box; represents the effective detection range of the manipulator, which is set to 80% of the maximum theoretical detection range;

[0115] (ii) Set the upper limit of the number of manipulators to 2 , and construct a candidate set of the number of manipulators, where . Calculate the volume coverage rate of the combined workspace of the manipulators with this number and the object to be detected for each candidate solution in the candidate set:

[0116]

[0117] In the formula, represents the volume coverage rate of the combined workspace of the manipulators with the number of and the object to be detected, represents the intersection volume of the combined workspace of the manipulators with the number of and the object to be detected;

[0118] (iii) Set a volume coverage rate threshold. On the condition that the volume coverage rate of the candidate solution is not less than the volume coverage rate threshold, select the candidate solution with the smallest value from the candidate set as the number of manipulators to perform the task.

[0119] In this embodiment, the volume coverage rate threshold is set to 99% to ensure full coverage of the object to be detected.

[0120] Optimization objective of the improved clustering algorithm It is expressed as:

[0121]

[0122] In the formula, is the preset number of clusters, which is equal to the number of robotic arms; represents the subset of detection points of the th cluster; represents the coordinates of the center point of the th cluster; represents the th detection point in the detection point set; represents the weight coefficient; represents the curvature distribution entropy, which quantifies the uniformity of the curvature distribution within the cluster and avoids concentrating high-curvature regions on a single robotic arm. Its calculation method is:

[0123]

[0124] In the formula, is the probability that the curvature of the detection points in cluster is in the th interval; is the number of intervals of the curvature histogram.

[0125] During the optimization process of the improved clustering algorithm, the following constraint conditions also need to be satisfied:

[0126] (1) The detection point must be within the working space of the robotic arm corresponding to the th cluster;

[0127] (2) The difference in task time consumption of each robotic arm does not exceed 20% of the average value;

[0128] (3) The distance between the motion trajectories of two robotic arms is greater than the safety distance , which is expressed as:

[0129]

[0130] In the formula, , respectively represent the positions where the ends of two different robotic arms are located at t time.

[0131] The specific process of optimizing the improved clustering algorithm is as follows: First, select initial clustering centers to ensure that each clustering center is located at within the common area of the workspace of the robotic arms; then perform iterative optimization. In the detection point allocation stage, allocate each detection point to the cluster that is reachable and has the minimum objective function; in the update stage, recalculate the cluster centers and constrain them to be within the workspace of the corresponding robotic arm; finally, perform post-processing, merge clusters with too few points to improve the calculation efficiency, and split clusters with too many points to avoid overloading a single robotic arm task.

[0132] Step S3: Use a sequential greedy search path planning algorithm with continuous optimization to plan an optimal path for each robotic arm to scan all the detection points in the corresponding subset of detection points.

[0133] Step S3 specifically includes the following steps:

[0134] Step S31: Use a sequential greedy search algorithm to sort the detection points that each robotic arm needs to execute to obtain an initial path.

[0135] The generation process of the initial path specifically includes the following steps: Given the current detection point , select the next best detection point from the subset of detection points such that the robotic arm can cover the most undetected area at the detection point and has the highest information collection efficiency.

[0136] The newly added information collection at any detection point position of the robotic arm is defined as follows:

[0137]

[0138] In the formula, represents the trace of the matrix; and represent the covariance of the object surface map after and before scanning at the detection point;

[0139] S32: Use the initial path points as decision variables and use a continuous optimization algorithm to optimize the initial path to obtain the optimal solution.

[0140] The initial path only contains multiple discrete detection points, but the ultimate goal of path planning is to plan a continuous path for the robotic arm. Therefore, it is necessary to further generate and optimize the path, and the goal of optimization is to maximize the time-averaged regional gain U of the planned path.

[0141] The generation process of the optimal path specifically includes the following steps: Connect the detection points on the initial trajectory using a polynomial trajectory, obtain the surface area measured along the polynomial trajectory, and optimize the polynomial trajectory with the maximization of the time-averaged regional gain U as the optimization objective to generate the optimal path. The time-averaged regional gainU Expressed as:

[0142]

[0143] In the formula, represents the surface area measured along the trajectory.

[0144] In step S3, the planned trajectory must be collision-free with the object to be detected. Therefore, constraints need to be added, specifically including:

[0145] First, construct an Euclidean signed distance field based on the surface mesh of the object to be detected, which represents the Euclidean distance from each point in space to its nearest obstacle. When performing the sequential greedy search algorithm in step S31, use the Euclidean signed distance field to add a constraint: the next detection point must be within the line of sight of the current detection point, which can provide preliminary collision avoidance for the continuous optimization in step S32. The line of sight is judged by sampling positions from the line segment connecting two detection points and checking whether any sampled position collides with the mesh of the object to be detected according to its Euclidean signed distance value. If there is a sampled position that collides with the mesh of the object to be detected, it is determined that the next detection point is not within the line of sight of the current detection point; otherwise, it is determined that the next detection point is within the line of sight of the current detection point.

[0146] In step S4, each robotic arm runs along the optimal path, performs image acquisition on the object to be detected at the detection points, and performs surface detection on the object to be detected using a multi-level defect detection method based on physical optical compensation and dynamic region growing.

[0147] Step S4 specifically includes the following steps:

[0148] In step S41, convert the vertices of the CAD model of the object to be detected from the base coordinate system to the camera coordinate system, remove the invisible triangular patches, and then map the three-dimensional space points to the two-dimensional image plane using the pinhole camera principle. Then, perform rasterization processing on the visible triangular patches using the scan line algorithm to generate a reference image of the object to be detected;

[0149] In step S42, establish an illumination compensation model to eliminate the difference between the actual illumination condition and the theoretical illumination model, perform illumination compensation on the acquired image to obtain a corrected image, and the illumination compensation model is expressed as:

[0150]

[0151]

[0152] In the formula, , represent the image pixel coordinates; represents the illumination compensation coefficient; represents the gray value of the actual image; represents the expected gray value of the image under the theoretical illumination model; represents a minimum value used to prevent division-by-zero errors; represents the diffuse reflection coefficient of the material of the object to be detected, with a value range ; represents the surface normal vector; represents the normalized light source direction vector, obtained through mirror sphere calibration; represents the specular reflection coefficient; represents the reflected light direction vector; represents the line-of-sight direction vector; represents the specular exponent used to control the sharpness of specular reflection;

[0153] For perform Gaussian filtering to eliminate noise and high-frequency fluctuations, obtaining :

[0154]

[0155] In the formula, represents the corrected illumination compensation coefficient; represents the Gaussian kernel used to retain the low-frequency non-uniformity of illumination; represents the convolution operation;

[0156] The corrected image is expressed as:

[0157]

[0158] The influence of uneven illumination has been eliminated in the corrected image, and only the reflection characteristics of the object surface, such as gray changes caused by defects, are retained.

[0159] In step S43, the cross-power spectrum of the reference image and the corrected image is calculated using the phase correlation method, and the cross-correlation matrix of the reference image and the corrected image is obtained through inverse Fourier transform, expressed as:

[0160]

[0161] In the formula, represents the cross-correlation matrix of the reference image and the corrected image; , represent the spatial frequency coordinate quantities in the frequency domain, represents the corrected image; represents the reference image; represents the two-dimensional Fourier transform that converts the image to the frequency domain; represents the complex conjugate operation used to calculate the frequency domain correlation; represents the inverse Fourier transform;

[0162] Step S44. Eliminate the positioning error of the robotic arm and adaptively distinguish noise from real defects through sub-pixel image registration and dynamic thresholding, which specifically includes the following processes:

[0163] Find the maximum peak coordinates from the cross-correlation matrix , perform quadratic fitting on the peak neighborhood to obtain the sub-pixel offset ;

[0164] Perform sub-pixel translation correction on the corrected image using bilinear interpolation to obtain the corrected image :

[0165]

[0166] Step S45. Extract the defect area from the actual image collected by the robotic arm, and use morphological processing to eliminate noise and connect real defect areas, which specifically includes the following steps:

[0167] Calculate the difference map between the corrected image and the reference image :

[0168]

[0169] Statistic the grayscale histogram of the difference map , take the average value and standard deviation of the grayscale histogram, set the threshold as , and the pixels exceeding are regarded as defects;

[0170] Perform closing operation on the difference map, use a 5x5 circular kernel structuring element for dilation followed by erosion to fill small holes; then perform opening operation, use a 5x5 circular kernel structuring element for erosion followed by dilation to eliminate isolated noise points, and then apply the region growing algorithm to accurately extract continuous defect areas. In the image after morphological processing, select the pixels with a grayscale difference exceeding 2 times the threshold as the growth starting point, and then check the neighboring pixels. If the grayscale value difference is less than a certain value, merge it into the growth region; the stopping condition is that the growth reaches the image boundary or 10 consecutive pixels no longer meet the merging condition.

[0171] Step S46. Perform quantitative analysis and classification on the defects through geometric parameter calculation and type discrimination rules.

[0172] The geometric parameter calculation includes the calculation of defect depth and defect area, where:

[0173] The calculation process of the defect depth is expressed as:

[0174]

[0175] Wherein, represents the number of pixels in the defect area; and respectively represent the gray values of the reference image and the actually acquired image at the pixel point ; is the depth conversion coefficient, obtained through calibration;

[0176] Defect area The calculation process is expressed as:

[0177]

[0178] Wherein, represents the image resolution, determined by the camera focal length and the working distance.

[0179] The types of defects are classified through discrimination rules:

[0180] Scratch determination is based on the relationship between the major axis length and the minor axis length of the defect area; specifically: if and the depth gradient direction is consistent with the major axis direction, it is determined as a scratch, is a ratio threshold set manually;

[0181] Pit determination is made through the defect area and the depth gradient; specifically: if and the depth gradient is radially distributed along the edge, it is determined as a pit, is an area threshold set manually;

[0182] Crack determination is based on the ratio of the actual perimeter of the defect area to the equivalent circle perimeter and the number of crack branches; specifically: if and the number of branches is greater than or equal to 3, it is determined as a crack, is a ratio threshold set manually.

[0183] Finally, statistical analysis is performed on the calculated geometric parameters to remove outliers, and a defect report is generated according to the type discrimination rules, including information such as the defect location, size, depth, and type.

[0184] S5: Generate a detection report based on the detection results of each robotic arm, and visually mark the defect area.

[0185] Step S5 specifically includes the following steps:

[0186] Step S51: Perform coordinate unification processing on the defect data uploaded by each robotic arm, convert the local detection coordinates to the global coordinate system, and generate a global defect point cloud set;

[0187] The conversion process of local detection coordinates to the global coordinate system is expressed as:

[0188]

[0189] In the formula, represents the coordinates of the defect point in the global coordinate system; represents the coordinates of the defect point in the local detection coordinate system; represents the real-time pose matrix of the end effector, which is calculated from the structural parameters of the robotic arm; represents the pose matrix of the base coordinate system of the robotic arm relative to the global coordinate system;

[0190] The global defect point cloud set is expressed as: , , in the formula, represents the total number of local detection points.

[0191] It should be noted that the global defect point cloud set also needs to remove duplicate points. The principle of removal is: calculate the Euclidean distance between any two defect points, set a distance threshold. If the Euclidean distance between two defect points is less than the distance threshold, then the two defect points are determined to be duplicate points, and one of them is removed and the other is retained.

[0192] Step S52: Determine the triangular patch to which each defect point in the global defect point cloud set belongs, and then perform RGB color coding on the triangular patch to which the defect point belongs according to the defect type to achieve defect annotation.

[0193] The mapping relationship between the defect point and the triangular patch is expressed as:

[0194]

[0195] In the formula, represents the triangular patch to which the defect point belongs; represents the operation of projecting the defect point onto the plane of the triangular patch; represents the set of triangular patches of the CAD model of the object to be detected; represents the k th triangular patch.

[0196] The RGB color coding rule is expressed as: the color coding for cracks is (255, 0, 0), the color coding for pits is (0, 0, 255), and the color coding for scratches is (0, 255, 0).

[0197] Generate a comprehensive report containing information such as the location, size, depth, and type of defects. The report supports users to filter defects of specific types or severities according to their needs and provides an interactive 3D view. Users can rotate and zoom the model to view defect details.

[0198] The embodiments of the present invention have been described above. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make many forms without departing from the spirit and scope protected by the claims of the present invention, and all of them fall within the protection scope of the present invention.

Claims

1. A method for detecting surface defects of three-dimensional objects based on multi-robot collaboration, characterized in that: The steps include: Step S1: Use the manifold Gaussian method to map the surface information field of the object to be detected and its mean function and covariance function to the manifold space, and use the Poisson disk sampling algorithm to sample and generate a detection point set in the manifold space. The "using the Poisson disk sampling algorithm to sample and generate a detection point set in the manifold space" specifically includes the following process: define a dynamic sampling density function, use the Poisson disk sampling algorithm based on the exclusion radius to sample in the manifold space, adjust the exclusion radius of the sampling point according to the dynamic sampling density function, and generate a detection point set in the space that meets the density constraint. The dynamic sampling density function is expressed as: Where, represents the sampling density; represents the basic sampling density; represents the curvature sensitivity coefficient; 、 They represent the principal curvatures of the triangular facets in the manifold space, which are obtained by surface differential geometry analysis methods; Step S2: Determine the workspace of each manipulator based on its structural parameters. Select the number of manipulators required to perform the task based on the volume coverage of the combined workspace of all manipulators and the object to be detected. An improved clustering algorithm based on traditional K-means clustering is introduced using curvature distribution entropy. This improved clustering algorithm is used to classify the detection point set into multiple detection point subsets in different clusters, and each manipulator is assigned a detection point subset. Step S3, using a sequential greedy search path planning algorithm for continuous optimization, to plan the optimal path for each robot arm to scan all the detection points in the corresponding detection point subset; In step S4, each robotic arm runs along the optimal path, collects images of the object to be inspected at the inspection point, and performs surface inspection on the object to be inspected using a multi-level defect detection method based on physical optical compensation and dynamic region growing; Step S5: Generate a test report based on the test results of each robot arm and visually mark the defective areas.

2. The method for detecting surface defects of three-dimensional objects based on multi-robot collaboration according to claim 1, characterized in that: In step S1, "using the manifold Gaussian method to map the surface information field of the object to be detected and its mean function and covariance function to the manifold space" specifically includes the following steps: A CAD model of the object to be detected is established, and the surface information field of the CAD model and its mean function and covariance function are mapped to the manifold space using the manifold Gaussian method. The kernel function used in the manifold space is the geodesic Matern3 / 2 kernel function, which is expressed as: Where, Represents the geodesic distance between any two points on the manifold space; represents the kernel function, which is used to characterize the spatial correlation between any two points in the manifold space; represents the variance of the surface information field; represents the length scale parameter.

3. The method for detecting surface defects of three-dimensional objects based on multi-robot collaboration according to claim 1, characterized in that: The workspace of the robotic arm is determined in the following way: the homogeneous transformation matrices of each joint in the robotic arm are multiplied together to obtain the pose matrix of the end effector relative to the base coordinate system; the angle range of each joint is then uniformly discretized, and the Monte Carlo random sampling method is used to generate a point cloud dataset of the reachable positions of the robotic arm end. The convex hull of the point cloud dataset is calculated, and the geometric boundary of the robotic arm workspace is extracted to obtain a polyhedron model representing the robotic arm workspace. The polyhedron model consists of a vertex set and a triangle facet set.

4. The method for detecting surface defects of three-dimensional objects based on multi-robot collaboration according to claim 3, characterized in that: The number of manipulators required to perform a task is determined as follows: (i) Quantify the size of the object to be detected using a bounding box, and calculate the minimum number of robotic arms based on the horizontal coverage requirements of the object to be detected. , the calculation process is expressed as: Where, Indicates rounding up; 、 Represent the length and width of the bounding box respectively; Indicates the effective detection range of the robot arm; (ii) Set the upper limit of the number of robotic arms to 2 , build a candidate set of the number of robotic arms ,in, , for each candidate solution in the candidate set, calculate the volume coverage of the joint workspace of the number of manipulators and the object to be detected. The calculation process is expressed as: Where, express The volume coverage of the joint workspace of the number of manipulators and the objects to be inspected; ; express The intersection volume of the joint workspace of the number of manipulators and the object to be detected; Indicates the total volume of the object to be detected; (iii) Set a volume coverage threshold, and select the candidate solution from the candidate set under the condition that the volume coverage of the candidate solution is not less than the volume coverage threshold. The candidate with the smallest value is selected as the number of robotic arms to perform the task.

5. The method for detecting surface defects of three-dimensional objects based on multi-robot collaboration according to claim 3, characterized in that: Improved optimization objectives of clustering algorithms Expressed as: Where, is the preset number of clusters, which is equal to the number of robotic arms; Indicates the A subset of detection points of a cluster; Indicates the The center coordinates of the clusters; Indicates the first detection points; represents the weight coefficient; represents the curvature distribution entropy, which is calculated as: Where, Cluster Inner detection point curvature In the The probability of an interval; is the number of curvature histogram intervals; The optimization process of the improved clustering algorithm is as follows: first select Initial cluster centers, ensuring that each cluster center is located The algorithm is then iterated and optimized. In the detection point allocation phase, each detection point is assigned to a cluster that is reachable and has the minimum objective function. In the update phase, the cluster centers are recalculated and constrained to be located within the workspace of the corresponding robotic arm; finally, post-processing is performed to merge clusters with too small a number of points and split clusters with too large a number of points.

6. The method for detecting surface defects of three-dimensional objects based on multi-robot collaboration according to claim 1, characterized in that: Step S3 specifically includes the following steps: Step S31, using a sequential greedy search algorithm to sort the detection points that each robot arm needs to execute, and obtain an initial path; The initial path generation process specifically includes the following steps: Given the current detection point , select the next best detection point from the detection point subset , so that the mechanical wall is at the detection point Able to cover the largest number of undetected areas and have the highest information collection efficiency; New information collection of the robot arm at any detection point is defined as follows: Where, represents the trace of the matrix; and represents the covariance of the object surface map before and after scanning at the detection point; Step S32: using the path points on the initial path as decision variables, a continuous optimization algorithm is used to optimize the initial path to obtain the optimal solution; The optimal path generation process specifically includes the following steps: using the polynomial trajectory to connect the detection points on the initial trajectory, obtaining the surface area measured along the polynomial trajectory, and using the time-averaged area gain U Maximize the optimization objective to optimize the polynomial trajectory to generate the optimal path, time average regional gain U Expressed as: Where, represents the surface area measured along the polynomial trajectory.

7. The method for detecting surface defects of three-dimensional objects based on multi-robot collaboration according to claim 6, characterized in that: In step S3, the planned trajectory needs to be constrained as follows: The next detection point must be within the line of sight of the current detection point. The line of sight is determined by sampling positions from the line segment connecting the two detection points and checking whether any sampled position collides with the mesh of the object to be detected based on its Euclidean signed distance value. If any sampled position collides with the mesh of the object to be detected, the next detection point is determined not to be within the line of sight of the current detection point; otherwise, the next detection point is determined to be within the line of sight of the current detection point.

8. The method for detecting surface defects of three-dimensional objects based on multi-robot collaboration according to claim 1, characterized in that: Step S4 specifically includes the following steps: Step S41: Convert the vertices of the CAD model of the object to be inspected from the base coordinate system to the camera coordinate system, remove the invisible triangles, and then use the pinhole camera principle to map the three-dimensional space points to the two-dimensional image plane. Then, use the scan line algorithm to rasterize the visible triangles to generate a reference image of the object to be inspected. Step S42: Establish an illumination compensation model to eliminate the difference between the actual illumination conditions and the theoretical illumination model, and perform illumination compensation on the captured image to obtain a corrected image. The illumination compensation model is expressed as: Where, 、 Represents the image pixel coordinates; Represents the illumination compensation coefficient; Represents the grayscale value of the actual image; Represents the expected grayscale value of the image under the theoretical illumination model; Indicates the minimum value, used to prevent division by zero errors; Indicates the diffuse reflectance coefficient of the material of the object to be detected, with a value range of ; represents the surface normal vector; Represents the normalized light source direction vector, obtained by specular sphere calibration; represents the specular reflection coefficient; Represents the direction vector of reflected light; Represents the sight direction vector; Represents the highlight index, which is used to control the sharpness of the specular reflection; right Perform Gaussian filtering to eliminate noise and high-frequency fluctuations, and obtain : Where, Indicates the corrected illumination compensation coefficient; represents the Gaussian kernel, which is used to preserve the low-frequency inhomogeneity of illumination; Represents the convolution operation; The corrected image is represented as: In step S43, the cross-power spectrum of the reference image and the corrected image is calculated using the phase correlation method, and the cross-correlation matrix of the reference image and the corrected image is obtained by inverse Fourier transform, which is expressed as: Where, represents the cross-correlation matrix between the reference image and the corrected image; 、 Represents the spatial frequency coordinate quantity in the frequency domain, represents the corrected image; represents the reference image; represents the two-dimensional Fourier transform, which converts the image into the frequency domain; represents the complex conjugate operation, which is used to calculate the frequency domain correlation; represents the inverse Fourier transform; Step S44, using sub-pixel image registration and dynamic thresholding to eliminate robot arm positioning errors and adaptively distinguish between noise and real defects, specifically includes the following process: From the cross-correlation matrix Find the maximum peak coordinates in , perform quadratic fitting on the peak neighborhood to obtain the sub-pixel offset ; Correct the image using bilinear interpolation Perform sub-pixel translation correction to obtain the corrected image : Step S45, extracting defect areas from the actual image captured by the robot arm, using morphological processing to eliminate noise and connect the real defect areas, specifically includes the following steps: Calculate the corrected image With reference image Difference graph of : Statistical difference plot Grayscale histogram, take the average value of the grayscale histogram and standard deviation , set the threshold to ,Exceed Pixels with 0 are considered defects; A closing operation is performed on the difference image, using a 5x5 circular kernel structure element for dilation and then erosion to fill small holes. An opening operation is then performed, using a 5x5 circular kernel structure element for erosion and then dilation to eliminate isolated noise points. The region growing algorithm is then applied to accurately extract continuous defect areas. In the morphologically processed image, pixels with a grayscale difference exceeding 2 times the threshold are selected as the growth starting point. The neighboring pixels are then checked. If the grayscale value difference is less than a certain value, they are merged into the growing region. The stopping condition is that the growth reaches the image boundary or 10 consecutive pixels no longer meet the merging condition.

9. The method for detecting surface defects of three-dimensional objects based on multi-robot collaboration according to claim 8, characterized in that: After step S45, the following steps are also included: Step S46: quantitatively analyze and classify the defects through geometric parameter calculation and type discrimination rules. The geometric parameter calculation includes the calculation of defect depth and defect area, where: Defect depth The calculation process is expressed as: Where, Indicates the number of pixels in the defect area; 、 Represents the reference image and the actual captured image after alignment at pixel points Gray value at ; is the depth conversion coefficient, obtained through calibration; Defect area The calculation process is expressed as: Where, Indicates image resolution, which is determined by the camera focal length and working distance; The types of defects are classified by the following discrimination rules: Scratch determination is based on the length of the main axis of the defect area and the secondary axis length Specifically: if If the depth gradient direction is consistent with the main axis direction, it is judged as a scratch. It is a ratio threshold set manually; Defect area of pits and depth gradient; specifically: if If the depth gradient is distributed radially along the edge, it is determined to be a pit. It is an artificially set area threshold; Crack determination is based on the actual perimeter of the defect area and the equivalent circumference The ratio of is determined by the number of crack branches; specifically: if If the number of branches is greater than or equal to 3, it is judged as a crack. It is a manually set ratio threshold.

10. The method for detecting surface defects of three-dimensional objects based on multi-robot collaboration according to claim 1, characterized in that: Step S5 specifically includes the following steps: Step S51: coordinate unification processing is performed on the defect data uploaded by each robot arm, and the local detection coordinates are converted into a global coordinate system to generate a global defect point cloud set; The conversion process from local detection coordinates to the global coordinate system is expressed as: Where, Indicates the coordinates of the defect point in the global coordinate system; Indicates the coordinates of the defect point in the local detection coordinate system; The real-time pose matrix of the end effector is calculated from the structural parameters of the robot arm; Represents the pose matrix of the manipulator base coordinate system relative to the global coordinate system; The global defect point cloud is represented as: , , where Indicates the total number of local detection points; Step S52: determine the triangular face to which each defect point in the global defect point cloud belongs, and then perform RGB color coding on the triangular face to which the defect point belongs according to the defect type to achieve defect labeling; The mapping relationship between defect points and triangles is expressed as: Where, Indicates the triangle to which the defect point belongs; Represents the operation of projecting the defect point onto the triangular face plane; A set of triangular facets of the CAD model of the object to be detected; Indicates the k triangles; The RGB color coding rule is as follows: the color coding of cracks is (255, 0, 0), the color coding of pits is (0, 0, 255), and the color coding of scratches is (0, 255, 0).

Images