Three-dimensional object surface defect detection method based on cooperation of multiple mechanical arms

Abstract

The invention provides a three-dimensional object surface defect detection method based on cooperation of multiple mechanical arms, and belongs to the technical field of defect detection. A series of detection points are generated according to a to-be-detected object CAD model and serve as position point coordinates of the mechanical arm for defect detection; then the number of the mechanical arms participating in the task is determined according to distribution of the detection points, and the detection points are grouped and distributed to all the mechanical arms; after distribution is completed, a continuously optimized sequential greedy search algorithm is used for conducting reasonable mechanical arm path planning, so that the mechanical arm can achieve full-coverage defect detection on the surface of the object; and finally, starting to execute a task, performing detection by utilizing a multi-stage defect detection method based on physical optical compensation and dynamic region growth according to image information acquired by the mechanical arms, generating a visual result report, and realizing efficient and accurate detection of surface defects of the three-dimensional object by virtue of cooperation of the multiple mechanical arms. And a reliable quality detection means is provided for industrial production.

Description

Technical Field

[0001] The present invention belongs to the technical field of defect detection, and particularly relates to a three-dimensional object surface defect detection method based on multi-robot arm cooperation. Background Art

[0002] Defect detection on the surface of an object is one of the key links to ensure product quality. Traditional surface defect detection methods mainly rely on manual visual inspection or single robot arm detection equipment. Manual visual inspection has many limitations, such as low detection efficiency, being easily affected by subjective factors resulting in low accuracy, and being prone to fatigue during long-term work, which is difficult to meet the needs of large-scale production; while single robot arm detection equipment, although improving the detection efficiency to a certain extent, due to the limitations of its movement range and detection perspective, there are many deficiencies in defect detection on the surface of complex three-dimensional objects, and it is difficult to comprehensively and accurately identify defects in various parts of the object.

[0003] With the continuous development of industrial automation and intelligence, multi-robot arm cooperation systems have gradually been applied to various complex tasks, showing great potential. Multi-robot arm cooperation can give full play to the advantages of each robot arm, and through reasonable task allocation and cooperative control, achieve efficient processing of complex environments and tasks. In the field of three-dimensional object surface defect detection, introducing the concept of multi-robot arm cooperation is expected to break through the bottleneck of traditional detection methods and achieve high-precision and high-efficiency defect detection. In the existing technology, although some studies involve the application of multi-robot arm cooperation in object recognition tasks, most of them focus on tasks such as object grasping and handling, and there is no application of multi-robot arm cooperation in three-dimensional object surface defect detection tasks.

[0004] Therefore, it is necessary to provide a three-dimensional object surface defect detection method based on multi-robot arm cooperation to solve the above problems. Summary of the Invention

[0005] The present invention provides a three-dimensional object surface defect detection method based on multi-robot arm cooperation. By optimizing the cooperative control strategy, detection path planning, and data processing and analysis process of multi-robot arms, it realizes efficient and accurate detection of three-dimensional object surface defects, provides a reliable quality detection means for industrial production, and can effectively solve at least one technical problem involved in the background art.

[0006] To solve the above technical problems, the present invention is implemented as follows: A three-dimensional object surface defect detection method based on multi-robot arm cooperation includes the following steps: Step S1, using the manifold Gaussian method to map the surface information field of the object to be detected, its mean function, and covariance function to the manifold space, and using the Poisson disk sampling algorithm to sample in the manifold space to generate a detection point set; Step S2: Determine the working space of each robotic arm according to its structural parameters. Select the number of robotic arms required to perform the task based on the volume coverage rate of the combined working space of all robotic arms and the object to be detected. On the basis of the traditional K-means clustering, introduce the curvature distribution entropy to form an improved clustering algorithm. Use the improved clustering algorithm to classify the detection point set into multiple subsets of detection points in different clusters, and assign a subset of detection points to each robotic arm. Step S3: Use the sequential greedy search path planning algorithm with continuous optimization to plan the optimal path for each robotic arm to scan all the detection points in the corresponding subset of detection points. Step S4: Each robotic arm runs according to the optimal path, collects images of the object to be detected at the detection points, and performs surface detection on the object to be detected using the multi-level defect detection method based on physical optical compensation and dynamic region growing. Step S5: Generate a detection report based on the detection results of each robotic arm, and visually mark the defect areas.

[0007] As a preferred improvement, Step S1 specifically includes the following steps: Step S11: Establish a CAD model of the object to be detected. Use the manifold Gaussian method to map the surface information field of the CAD model and its mean function and covariance function to the manifold space. The kernel function used in the manifold space is the geodesic Matern 3 / 2 kernel function, which is expressed as: In the formula, represents the geodesic distance between any two points in the manifold space; represents the kernel function, which is used to characterize the spatial correlation between any two points in the manifold space; represents the variance of the surface information field; represents the length scale parameter; Step S12: Define the dynamic sampling density function. Use the Poisson disk sampling algorithm based on the rejection radius to perform sampling in the manifold space. Adjust the rejection radius of the sampling points according to the dynamic sampling density function, and generate a detection point set that satisfies the density constraint in the space. The dynamic sampling density function is expressed as: In the formula, represents the sampling density; represents the basic sampling density; represents the curvature sensitivity coefficient; and respectively represent the principal curvatures of the triangular patches in the manifold space, which are obtained through the surface differential geometry analysis method.

[0008] As a preferred improvement, the working space of the robotic arm is determined as follows: Multiply the homogeneous transformation matrices of each joint in the robotic arm to obtain the pose matrix of the end effector relative to the base coordinate system; then uniformly discretize the angular ranges of each joint, use the Monte Carlo random sampling method to generate a point cloud dataset of the reachable positions of the robotic arm end, perform a convex hull calculation on the point cloud dataset, extract the geometric boundary of the robotic arm working space, and obtain a polyhedron model representing the robotic arm working space. The polyhedron model consists of a vertex set and a triangular facet set.

[0009] As a preferred improvement, the number of robotic arms required to perform a task is determined as follows: (i) Quantify the size of the object to be detected through a bounding box, and calculate the minimum number of robotic arms based on the coverage requirement of the object to be detected in the horizontal direction , and the calculation process is expressed as: In the formula, represents rounding up; , respectively represent the length and width of the bounding box; represents the effective detection range of the robotic arm; (ii) Set the upper limit of the number of robotic arms to 2 , and construct a candidate set of the number of robotic arms , where , and calculate the volume coverage rate of the combined working space of the robotic arms with this number and the object to be detected for each candidate solution in the candidate set. The calculation process is expressed as: In the formula, represents the volume coverage rate of the combined working space of the robotic arms with the number of ; represents the intersection volume of the combined working space of the robotic arms with the number of and the object to be detected; (iii) Set a volume coverage rate threshold, and select the candidate solution with the smallest value from the candidate set as the number of robotic arms to perform the task on the condition that the volume coverage rate of the candidate solution is not less than the volume coverage rate threshold.

[0010] As a preferred improvement, the optimization objective of the improved clustering algorithm is expressed as: In the formula, is the preset number of clusters, which is equal to the number of robotic arms; Represents the subset of detection points of the th cluster; Represents the central point coordinates of the th cluster; Represents the th detection point in the detection point set; Represents the weight coefficient; Represents the curvature distribution entropy, and its calculation method is: In the formula, is the curvature of the detection points in cluster in the th interval probability; Is the number of intervals of the curvature histogram; The specific process of optimizing the improved clustering algorithm is as follows: First, select initial clustering centers to ensure that each clustering center is located within the common area of the workspace of the robotic arm; Then perform iterative optimization. In the detection point assignment stage, assign each detection point to the cluster that is reachable and has the minimum objective function; In the update stage, recalculate the clustering center and constrain it to be within the workspace of the corresponding robotic arm; Finally, perform post-processing to merge clusters with too few points and split clusters with too many points.

[0011] As a preferred improvement, step S3 specifically includes the following steps: Step S31, use the sequential greedy search algorithm to sort the detection points required for each robotic arm to obtain the initial path; The generation process of the initial path specifically includes the following steps: Given the current detection point , select the next best detection point from the subset of detection points, so that the robotic arm can cover the most undetected areas at the detection point and has the highest information collection efficiency; The definition of the new information collection at any detection point position of the robotic arm is as follows: In the formula, represents the trace of the matrix; and represent the covariance of the object surface map after and before scanning at the detection point; Step S32, use the continuous optimization algorithm to optimize the initial path with the path points on the initial path to obtain the optimal solution; The generation process of the optimal path specifically includes the following steps: Connect the detection points on the initial trajectory using a polynomial trajectory, obtain the surface area measured along the polynomial trajectory, and optimize the polynomial trajectory with the time-averaged area gain U maximized as the optimization objective to generate the optimal path, and the time-averaged area gain U is expressed as: where represents the surface area measured along the polynomial trajectory.

[0012] As a preferred improvement, in step S3, the following constraints need to be added to the planned trajectory: The next detection point must be within the line of sight of the current detection point. Among them, the line of sight is judged by sampling positions from the line segment connecting two detection points and checking whether any sampled position collides with the mesh of the object to be detected according to its Euclidean signed distance value. If there is a sampled position that collides with the mesh of the object to be detected, it is determined that the next detection point is not within the line of sight of the current detection point; otherwise, it is determined that the next detection point is within the line of sight of the current detection point.

[0013] As a preferred improvement, step S4 specifically includes the following steps: Step S41, convert the vertices of the CAD model of the object to be detected from the base coordinate system to the camera coordinate system, remove the invisible triangular patches, and then map the three-dimensional space points to the two-dimensional image plane using the pinhole camera principle. Then, rasterize the visible triangular patches using the scan line algorithm to generate the reference image of the object to be detected; Step S42, establish a light compensation model to eliminate the difference between the actual light condition and the theoretical light model, and perform light compensation on the collected image to obtain the corrected image. The light compensation model is expressed as: where and represent the image pixel coordinates; represents the light compensation coefficient; represents the gray value of the actual image; represents the expected gray value of the image under the theoretical light model; represents the minimum value, used to prevent division by zero errors; represents the diffuse reflection coefficient of the material of the object to be detected, and the value range is ; represents the surface normal vector; represents the normalized light source direction vector, obtained through specular sphere calibration; represents the specular reflection coefficient; Represents the reflected light direction vector; Represents the line-of-sight direction vector; Represents the specular exponent, used to control the sharpness of specular reflection; For Perform Gaussian filtering to eliminate noise and high-frequency fluctuations, obtaining : In the formula, Represents the corrected illumination compensation coefficient; Represents the Gaussian kernel, used to retain the low-frequency non-uniformity of illumination; Represents the convolution operation; The corrected image is expressed as: In step S43, the cross-power spectrum of the reference image and the corrected image is calculated using the phase correlation method, and the cross-correlation matrix of the reference image and the corrected image is obtained through inverse Fourier transform, expressed as: In the formula, Represents the cross-correlation matrix of the reference image and the corrected image; , Represents the spatial frequency coordinate quantity in the frequency domain, Represents the calibrated image; Represents the reference image; Represents the two-dimensional Fourier transform, which converts the image to the frequency domain; Represents the complex conjugate operation, used to calculate the frequency domain correlation; Represents the inverse Fourier transform; In step S44, the positioning error of the robotic arm is eliminated and noise and real defects are adaptively distinguished through sub-pixel image registration and dynamic thresholding, specifically including the following process: Find the maximum peak coordinate from the cross-correlation matrix , perform quadratic fitting on the peak neighborhood to obtain the sub-pixel offset ; Use bilinear interpolation to perform sub-pixel translation correction on the corrected image to obtain the calibrated image : In step S45, the defect area of the actual image collected by the robotic arm is extracted, and morphological processing is used to eliminate noise and connect real defect areas, specifically including the following steps: Calculate the difference image between the calibrated image and the reference image : Statistical difference map The grayscale histogram of is taken, and the average value of the grayscale histogram is taken , and the threshold is set to , and the pixels exceeding are regarded as defects; Perform a closing operation on the difference map, dilate and then erode using a 5x5 circular kernel structural element to fill small holes; then perform an opening operation, erode and then dilate using a 5x5 circular kernel structural element to eliminate isolated noise points, and then apply the region growing algorithm to accurately extract the continuous defect region. In the morphologically processed image, select the pixels with a grayscale difference exceeding twice the threshold as the growth starting point, and then check the neighboring pixels. If the grayscale value difference is less than a certain value, merge them into the growth region; the stop condition is that the growth reaches the image boundary or 10 consecutive pixels no longer meet the merging condition.

[0014] As a preferred improvement, after step S45, the following steps are further included: Step S46, perform quantitative analysis and classification on the defects through geometric parameter calculation and type discrimination rules. Geometric parameter calculation includes the calculation of defect depth and defect area, where: Defect depth The calculation process is expressed as: In the formula, represents the number of pixels in the defect region; , respectively represent the grayscale values of the reference image and the actually acquired image after alignment at the pixel point ; is the depth conversion coefficient, obtained through calibration; Defect area The calculation process is expressed as: In the formula, represents the image resolution, determined by the camera focal length and the working distance; The type of the defect is classified through the discrimination rule: Scratch determination is based on the relationship between the major axis length and the minor axis length of the defect region; specifically: if and the depth gradient direction is consistent with the major axis direction, it is determined as a scratch, is the ratio threshold set manually; Pit determination is determined by the defect area and the depth gradient; specifically: if If the depth gradient is radially distributed along the edge, it is determined as a pit. is the area threshold set manually; The crack determination is based on the actual perimeter of the defect area and the perimeter of the equivalent circle The ratio and the number of crack branches are used to determine; specifically: if and the number of branches is greater than or equal to 3, it is determined as a crack. is the ratio threshold set manually.

[0015] As a preferred improvement, step S5 specifically includes the following steps: Step S51, perform coordinate unification processing on the defect data uploaded by each robotic arm, convert the local detection coordinates to the global coordinate system, and generate a global defect point cloud set; The conversion process of local detection coordinates to the global coordinate system is expressed as: In the formula, represents the coordinates of the defect point in the global coordinate system; represents the coordinates of the defect point in the local detection coordinate system; represents the real-time pose matrix of the end effector, which is calculated from the structural parameters of the robotic arm; represents the pose matrix of the robotic arm base coordinate system relative to the global coordinate system; The global defect point cloud set is expressed as: , , in the formula, represents the total number of local detection points; Step S52, determine the triangular patch to which each defect point in the global defect point cloud set belongs, and then perform RGB color coding on the triangular patch to which the defect point belongs according to the defect type to achieve defect annotation; The mapping relationship between the defect point and the triangular patch is expressed as: In the formula, represents the triangular patch to which the defect point belongs; represents the operation of projecting the defect point onto the triangular patch plane; is the triangular patch set of the CAD model of the object to be detected; represents the k th triangular patch; The RGB color coding rule is expressed as: the color coding of the crack is (255, 0, 0), the color coding of the pit is (0, 0, 255), and the color coding of the scratch is (0, 255, 0).

[0016] The beneficial effects of the present invention are as follows: (1) Through the dynamic task allocation and collaborative scheduling mechanism of the present invention, multiple robotic arms can participate in the defect detection task simultaneously. Compared with the traditional single-robotic arm solution, it can significantly improve the task execution efficiency, greatly reduce the detection blind area, and increase the coverage rate. (2) The present invention adopts a continuously optimized sequential greedy search path planning algorithm, combines the mean and covariance information of the surface of the object to be detected, reduces the repeated coverage rate of the robotic arm's running path, and simultaneously constructs an Euclidean signed distance field to dynamically adjust the safety distance threshold to achieve a collision-free trajectory, reducing the collision risk. (3) The present invention adopts a multi-level detection method based on physical optical compensation and dynamic region growing. By light modeling, it eliminates the reflection interference on the surface of the object, realizes high-precision sub-pixel registration, reduces the false detection rate, and at the same time combines morphological processing to effectively suppress noise interference and improve the robustness of defect detection on the surface of complex curved surfaces. Specific implementation mode

[0017] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0018] This implementation mode provides a three-dimensional object surface defect detection method based on multi-robotic arm cooperation, including the following steps: Step S1, use the manifold Gaussian method to map the surface information field of the object to be detected and its mean function and covariance function to the manifold space, and use the Poisson disk sampling algorithm to sample in the manifold space to generate a detection point set.

[0019] Step S1 specifically includes the following steps: Step S11, establish a CAD model of the object to be detected, use the manifold Gaussian method to map the surface information field of the CAD model and its mean function and covariance function to the manifold space, and the kernel function adopted by the manifold space is the geodesic Matern3 / 2 kernel function, expressed as: In the formula, represents the geodesic distance between any two points in the manifold space; represents the kernel function, which is used to characterize the spatial correlation between any two points in the manifold space; represents the variance of the surface information field; represents the length scale parameter; Step S12: Define a dynamic sampling density function. Use the Poisson disk sampling algorithm based on the rejection radius to sample in the manifold space, adjust the rejection radius of the sampling points according to the dynamic sampling density function, and generate a set of detection points that meet the density constraints in the space. The dynamic sampling density function is expressed as: In the formula, represents the sampling density; represents the basic sampling density; represents the curvature sensitivity coefficient; , respectively represent the principal curvatures of the triangular facets in the manifold space, which are obtained by the surface differential geometry analysis method.

[0020] The setting of the dynamic sampling density function can ensure a higher sampling density in the key areas (high-curvature areas), while avoiding uneven or repeated distribution of detection points.

[0021] Step S2: Determine the workspace of each robotic arm according to the structural parameters of the robotic arm. Select the number of robotic arms required to perform the task based on the volume coverage rate of the combined workspace of all robotic arms and the object to be detected. Introduce the curvature distribution entropy on the basis of the traditional K-means clustering to form an improved clustering algorithm. Use the improved clustering algorithm to classify the set of detection points into multiple subsets of detection points in different clusters, and assign a subset of detection points to each robotic arm.

[0022] The workspace of the robotic arm is determined as follows: Multiply the homogeneous transformation matrices of each joint in the robotic arm to obtain the pose matrix of the end effector relative to the base coordinate system; Then uniformly discretize the angle ranges of each joint, and use the Monte Carlo random sampling method to generate a point cloud dataset of the reachable positions of the robotic arm end. Perform a convex hull calculation on the point cloud dataset , extract the geometric boundary of the robotic arm workspace, and obtain a polyhedron model representing the robotic arm workspace. The polyhedron model consists of a vertex set and a triangular facet set .

[0023] The Monte Carlo random sampling method is a sampling method based on probability statistics and random simulation. In the analysis of the robotic arm workspace, the Monte Carlo method randomly samples the angle combinations of each joint of the robotic arm, calculates the position of the end effector, and finally determines the reachable workspace range through the point cloud distribution.

[0024] The number of robotic arms required to perform the task is determined as follows: (i) Quantify the size of the object to be detected through a bounding box, and calculate the minimum number of robotic arms based on the coverage requirement of the object to be detected in the horizontal direction. The calculation process is expressed as: In the formula, represents rounding up; , represent the length and width of the bounding box respectively; represents the effective detection range of the robotic arm, which is set to 80% of the maximum theoretical detection range; (ii) Set the upper limit of the number of robotic arms to 2 , and construct a candidate set of the number of robotic arms , where , calculate the volume coverage rate of the combined working space of the robotic arms with this number and the object to be detected for each candidate solution in the candidate set: In the formula, represents the volume coverage rate of the combined working space of the robotic arms with the number of and the object to be detected, represents the intersection volume of the combined working space of the robotic arms with the number of and the object to be detected; represents the total volume of the object to be detected; (iii) Set a volume coverage rate threshold. On the condition that the volume coverage rate of the candidate solution is not less than the volume coverage rate threshold, select the candidate solution with the smallest value from the candidate set as the number of robotic arms to perform the task.

[0025] In this embodiment, the volume coverage rate threshold is set to 99% to ensure full coverage of the object to be detected.

[0026] Optimization objective of the improved clustering algorithm is expressed as: In the formula, is the preset number of clusters, which is equal to the number of robotic arms; represents the subset of detection points of the -th cluster; represents the center point coordinates of the -th cluster; represents the -th detection point in the detection point set; represents the weight coefficient; Denote the curvature distribution entropy, which quantifies the uniformity of the curvature distribution within the cluster and avoids concentrating high-curvature regions to a single robotic arm. Its calculation method is as follows: In the formula, is the cluster the curvature of the detection point within in the th interval; is the number of intervals of the curvature histogram.

[0027] During the optimization process of the improved clustering algorithm, the following constraint conditions also need to be satisfied: (1) The detection point must be located within the workspace of the robotic arm corresponding to the th cluster; The difference in task time consumption of each robotic arm does not exceed 20% of the average value; (3) The distance between the motion trajectories of two robotic arms is greater than the safety distance , expressed as: In the formula, , respectively represent t the positions of the ends of two different robotic arms at time

[0028] The optimization process of the improved clustering algorithm is specifically as follows: First, select initial clustering centers to ensure that each clustering center is located within the common area of the workspaces of robotic arms; then perform iterative optimization. In the detection point assignment stage, assign each detection point to the cluster that is reachable and has the minimum objective function; in the update stage, recalculate the clustering centers and constrain them to be within the workspaces of the corresponding robotic arms; finally, perform post-processing, merge clusters with too few points to improve the calculation efficiency, and split clusters with too many points to avoid overloading a single robotic arm with tasks.

[0029] Step S3, adopt the sequential greedy search path planning algorithm with continuous optimization to plan the optimal path for each robotic arm to scan all detection points in the corresponding subset of detection points.

[0030] Step S3 specifically includes the following steps: Step S31, use the sequential greedy search algorithm to sort the detection points that each robotic arm needs to execute to obtain the initial path.

[0031] The generation process of the initial path specifically includes the following steps: Given the current detection point , select the next best detection point , enabling the robotic wall to cover the most undetected areas at the detection points and having the highest information collection efficiency.

[0032] The newly added information collection of the robotic arm at any detection point position is defined as follows: In the formula, represents the trace of the matrix; and represent the covariance of the object surface maps after and before scanning at the detection point; S32: Using the path points on the initial path as decision variables, a continuous optimization algorithm is used to optimize the initial path to obtain the optimal solution.

[0033] The initial path only contains multiple discrete detection points, but the ultimate goal of path planning is to plan a continuous path for the robotic arm. Therefore, it is necessary to further generate and optimize the path, and the optimization goal is to maximize the time-average regional gain U of the planned path.

[0034] The generation process of the optimal path specifically includes the following steps: Connect the detection points on the initial trajectory using a polynomial trajectory, obtain the surface area measured along the polynomial trajectory, and optimize the polynomial trajectory with the maximization of the time-average regional gain U as the optimization goal to generate the optimal path. The time-average regional gain U is expressed as: In the formula, represents the surface area measured along the trajectory.

[0035] In step S3, the planned trajectory must be collision-free with the object to be detected. Therefore, constraints need to be added, specifically including: First, construct an Euclidean signed distance field based on the surface mesh of the object to be detected, which represents the Euclidean distance from each point in space to its nearest obstacle. When executing the sequential greedy search algorithm in step S31, use the Euclidean signed distance field to add a constraint: the next detection point must be within the line of sight of the current detection point, which can provide preliminary collision avoidance for the continuous optimization in step S32. The line of sight is judged by sampling positions from the line segment connecting two detection points and checking whether any sampled position collides with the mesh of the object to be detected according to its Euclidean signed distance value. If there is a sampled position that collides with the mesh of the object to be detected, it is determined that the next detection point is not within the line of sight of the current detection point; otherwise, it is determined that the next detection point is within the line of sight of the current detection point.

[0036] In step S4, each robotic arm operates along the optimal path, collects images of the object to be detected at the detection points, and performs surface detection on the object to be detected using a multi-level defect detection method based on physical optical compensation and dynamic region growth.

[0037] Step S4 specifically includes the following steps: In step S41, the vertices of the CAD model of the object to be detected are transformed from the base coordinate system to the camera coordinate system. After removing the invisible triangular patches, the three-dimensional space points are mapped to the two-dimensional image plane using the pinhole camera principle. Then, the visible triangular patches are rasterized using the scan-line algorithm to generate a reference image of the object to be detected. In step S42, an illumination compensation model is established to eliminate the difference between the actual illumination conditions and the theoretical illumination model. The collected image is subjected to illumination compensation to obtain a corrected image. The illumination compensation model is expressed as: In the formula, and represent the image pixel coordinates; represents the illumination compensation coefficient; represents the gray value of the actual image; represents the expected gray value of the image under the theoretical illumination model; represents a minimum value used to prevent division-by-zero errors; represents the diffuse reflection coefficient of the material of the object to be detected, with a value range of ; represents the surface normal vector; represents the normalized light source direction vector obtained through mirror sphere calibration; represents the specular reflection coefficient; represents the reflected light direction vector; represents the line-of-sight direction vector; represents the specular highlight exponent used to control the sharpness of specular reflection; Perform Gaussian filtering on to eliminate noise and high-frequency fluctuations, obtaining : In the formula, represents the corrected illumination compensation coefficient; represents the Gaussian kernel used to retain the low-frequency non-uniformity of illumination; represents the convolution operation; The corrected image is expressed as: The corrected image has eliminated the influence of uneven illumination and only retains the surface reflection characteristics of the object, such as gray value changes caused by defects.

[0038] Step S43: Calculate the cross-power spectrum of the reference image and the corrected image using the phase correlation method, and obtain the cross-correlation matrix of the reference image and the corrected image through inverse Fourier transform, which is expressed as: In the formula, represents the cross-correlation matrix of the reference image and the corrected image; and represent the spatial frequency coordinate quantities in the frequency domain, represents the corrected image; represents the reference image; represents the two-dimensional Fourier transform, which converts the image to the frequency domain; represents the complex conjugate operation, which is used to calculate the frequency domain correlation; represents the inverse Fourier transform; Step S44: Eliminate the positioning error of the robotic arm and adaptively distinguish noise from real defects through sub-pixel image registration and dynamic threshold, which specifically includes the following process: Find the maximum peak coordinate from the cross-correlation matrix , perform quadratic fitting on the peak neighborhood to obtain the sub-pixel offset ; Use bilinear interpolation to perform sub-pixel translation correction on the corrected image to obtain the corrected image : Step S45: Extract the defect area from the actual image collected by the robotic arm, and use morphological processing to eliminate noise and connect real defect areas, which specifically includes the following steps: Calculate the difference image between the corrected image and the reference image : Statistical gray histogram of the difference image , take the average value and standard deviation of the gray histogram, set the threshold to , and the pixels exceeding are regarded as defects; Perform a closing operation on the difference image, dilate and then erode it using a 5x5 circular kernel structuring element to fill small holes; then perform an opening operation, erode and then dilate it using a 5x5 circular kernel structuring element to eliminate isolated noise points, and then apply a region growing algorithm to accurately extract the continuous defect region. In the morphologically processed image, select the pixels with a gray level difference exceeding twice the threshold as the growth starting point, and then check the neighboring pixels. If the gray level difference is less than a certain value, merge them into the growth region; the stopping condition is that the growth reaches the image boundary or 10 consecutive pixels no longer meet the merging condition.

[0039] Step S46, perform quantitative analysis and classification on the defects through geometric parameter calculation and type discrimination rules.

[0040] The geometric parameter calculation includes the calculation of the defect depth and the defect area, where: Defect depth The calculation process is expressed as: In the formula, represents the number of pixels in the defect region; , respectively represent the gray levels of the reference image and the actually acquired image at the pixel point ; is the depth conversion coefficient, obtained through calibration; Defect area The calculation process is expressed as: In the formula, represents the image resolution, determined by the camera focal length and the working distance.

[0041] The type of the defect is classified through the discrimination rules: Scratch determination is based on the relationship between the major axis length and the minor axis length of the defect region; specifically: if and the depth gradient direction is consistent with the major axis direction, it is determined as a scratch, is the ratio threshold set manually; Pit determination is made through the defect area and the depth gradient; specifically: if and the depth gradient is radially distributed along the edge, it is determined as a pit, is the area threshold set manually; Crack determination is based on the ratio of the actual perimeter of the defect region to the equivalent circle perimeter and the number of crack branches; specifically: if If the number of branches is greater than or equal to 3, it is determined as a crack. is a ratio threshold set manually.

[0042] Finally, statistical analysis is performed on the calculated geometric parameters to remove outliers, and a defect report is generated according to the type discrimination rule, including information such as defect location, size, depth, and type.

[0043] S5: Generate an inspection report based on the inspection results of each robotic arm, and visually annotate the defect area.

[0044] Step S5 specifically includes the following steps: Step S51, perform coordinate unification processing on the defect data uploaded by each robotic arm, convert the local inspection coordinates to the global coordinate system, and generate a global defect point cloud set; The conversion process of local inspection coordinates to the global coordinate system is expressed as: In the formula, represents the coordinates of the defect point in the global coordinate system; represents the coordinates of the defect point in the local inspection coordinate system; represents the real-time pose matrix of the end effector, which is calculated from the structural parameters of the robotic arm; represents the pose matrix of the robotic arm base coordinate system relative to the global coordinate system; The global defect point cloud set is expressed as: , , in the formula, represents the total number of local detection points.

[0045] It should be noted that the global defect point cloud set also needs to remove duplicate points. The principle of removal is: calculate the Euclidean distance between any two defect points, set a distance threshold. If the Euclidean distance between two defect points is less than the distance threshold, then the two defect points are determined as duplicate points, and one of them is removed and the other is retained.

[0046] Step S52, determine the triangular patch to which each defect point in the global defect point cloud set belongs, and then perform RGB color coding on the triangular patch to which the defect point belongs according to the defect type to achieve defect annotation.

[0047] The mapping relationship between the defect point and the triangular patch is expressed as: In the formula, represents the triangular patch to which the defect point belongs; represents the operation of projecting the defect point onto the triangular patch plane; represents the set of triangular patches of the CAD model of the object to be detected; represents thek triangular patches.

[0048] The RGB color coding rule is expressed as: the color coding of cracks is (255, 0, 0), the color coding of pits is (0, 0, 255), and the color coding of scratches is (0, 255, 0).

[0049] Generate a comprehensive report containing information such as the location, size, depth, and type of defects. The report supports users to filter specific types or severities of defects according to their needs and provides an interactive 3D view. Users can rotate and zoom the model to view defect details.

[0050] The embodiments of the present invention have been described above, but the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make many forms without departing from the purpose of the present invention and the scope protected by the claims, and all of them belong to the protection scope of the present invention.

Claims

1. A three-dimensional object surface defect detection method based on multi-robot arm cooperation, characterized in that It includes the following steps: Step S1: Use the manifold Gaussian method to map the surface information field, its mean function, and covariance function of the object to be detected into the manifold space, and use the Poisson disk sampling algorithm to sample in the manifold space to generate a detection point set; Step S2: Determine the working space of each robotic arm according to the structural parameters of the robotic arm. Select the number of robotic arms required to perform the task based on the volume coverage rate of the combined working space of all robotic arms and the object to be detected. Introduce curvature distribution entropy on the basis of the traditional K-means clustering to form an improved clustering algorithm. Use the improved clustering algorithm to classify the detection point set into multiple subsets of detection points in different clusters, and assign a subset of detection points to each robotic arm; Step S3: Use the sequential greedy search path planning algorithm with continuous optimization to plan the optimal path for each robotic arm to scan all detection points in the corresponding subset of detection points; Step S4: Each robotic arm runs according to the optimal path, collects images of the object to be detected at the detection points, and performs surface detection on the object to be detected using a multi-level defect detection method based on physical optical compensation and dynamic region growing; Step S5: Generate a detection report based on the detection results of each robotic arm, and visually annotate the defect areas.

2. The three-dimensional object surface defect detection method based on multi-robot arm cooperation according to claim 1, characterized in that Step S1 specifically includes the following steps: Step S11: Establish a CAD model of the object to be detected, and use the manifold Gaussian method to map the surface information field, its mean function, and covariance function of the CAD model into the manifold space. The kernel function used in the manifold space is the geodesic Matern 3 / 2 kernel function, which is expressed as: In the formula, represents the geodesic distance between any two points on the manifold space; represents the kernel function, which is used to characterize the spatial correlation between any two points on the manifold space; represents the variance of the surface information field; represents the length scale parameter; Step S12: Define a dynamic sampling density function, use the Poisson disk sampling algorithm based on the rejection radius to sample in the manifold space, adjust the rejection radius of the sampling points according to the dynamic sampling density function, and generate a detection point set that satisfies the density constraint in the space. The dynamic sampling density function is expressed as: In the formula, represents the sampling density; represents the basic sampling density; represents the curvature sensitivity coefficient; , respectively represent the principal curvatures of the triangular patches in the manifold space, which are obtained by the surface differential geometry analysis method.

3. The three-dimensional object surface defect detection method based on multi-robot arm cooperation according to claim 1, characterized in that The working space of the robotic arm is determined in the following way: Multiply the homogeneous transformation matrices of each joint in the robotic arm to obtain the pose matrix of the end effector relative to the base coordinate system; then uniformly discretize the angle ranges of each joint, use the Monte Carlo random sampling method to generate a point cloud data set of the reachable positions of the robotic arm end, perform a convex hull calculation on the point cloud data set, extract the geometric boundary of the robotic arm working space, and obtain a polyhedron model representing the robotic arm working space. The polyhedron model consists of a vertex set and a triangular facet set.

4. The three-dimensional object surface defect detection method based on multi-robot arm cooperation according to claim 3, characterized in that, The number of robotic arms required to perform the task is determined in the following way: (i) Quantify the size of the object to be detected through a bounding box, and calculate the minimum number of robotic arms based on the coverage requirements of the object to be detected in the horizontal direction. , and the calculation process is expressed as: In the formula, represents rounding up; , respectively represent the length and width of the bounding box; represents the effective detection range of the robotic arm; (ii) Set the upper limit of the number of robotic arms to 2 , construct a candidate set for the number of robotic arms , where , for each candidate solution in the candidate set, calculate the volume coverage rate of the combined workspace of the robotic arms of this number and the object to be detected. The calculation process is expressed as: In the formula, represents the volume coverage rate of the combined workspace of the robotic arms and the object to be detected; ; represents the intersection volume of the combined workspace of the robotic arms and the object to be detected; represents the total volume of the object to be detected; (iii)Set a volume coverage rate threshold, and select the candidate solution with the smallest value as the number of robotic arms to perform the task on the condition that the volume coverage rate of the candidate solution is not less than the volume coverage rate threshold from the candidate set .

5. The three-dimensional object surface defect detection method based on multi-robot arm cooperation according to claim 3, characterized in that Optimization Objectives of Improved Clustering Algorithms It is expressed as: Wherein, is the preset number of clusters, which is equal to the number of robotic arms; represents the subset of detection points of the th cluster; represents the coordinates of the center point of the th cluster; represents the th detection point in the detection point set; represents the weight coefficient; represents the curvature distribution entropy, and its calculation method is: In the formula, is the cluster curvature of the detection point in the probability of the interval; is the number of intervals of the curvature histogram; The specific process of improving the clustering algorithm optimization is as follows: First, select initial clustering centers to ensure that each clustering center is located within the common area of the working space of the robotic arms; then perform iterative optimization. In the detection point assignment stage, assign each detection point to the cluster that is reachable and has the minimum objective function. In the update stage, recalculate the cluster centers and constrain them to be within the working space of the corresponding robotic arms; finally, perform post-processing to merge clusters with too few points and split clusters with too many points.

6. The three-dimensional object surface defect detection method based on multi-robot cooperation according to claim 1, characterized in that Step S3 specifically includes the following steps: Step S31: Use the sequential greedy search algorithm to sort the detection points that each robotic arm needs to execute to obtain an initial path; The generation process of the initial path specifically includes the following steps: Given the current detection point , select the next best detection point from the subset of detection points , so that the robotic wall can cover the most undetected areas at the detection point and has the highest information collection efficiency; New information collection for the robotic arm at any detection point location is defined as follows: wherein, represents the trace of a matrix; and represent the covariance of the object surface maps after and before scanning at the detection points; Step S32: Use the continuous optimization algorithm to optimize the initial path with the path points on the initial path as decision variables to obtain the optimal solution; The generation process of the optimal path specifically includes the following steps: connecting the detection points on the initial trajectory with a polynomial trajectory, obtaining the surface area measured along the polynomial trajectory, and optimizing the polynomial trajectory with the maximization of the time-averaged regional gain U as the optimization objective to generate the optimal path, and the time-averaged regional gain U is expressed as: In the formula, represents the surface area measured along the polynomial trajectory.

7. The three-dimensional object surface defect detection method based on multi-robot arm cooperation according to claim 6, characterized in that In Step S3, the following constraints need to be added to the planned trajectory: The next detection point must be within the line of sight of the current detection point. Here, the line of sight is determined by sampling positions from the line segment connecting the two detection points and checking whether any sampled position collides with the mesh of the object to be detected according to its Euclidean signed distance value. If there is a sampled position that collides with the mesh of the object to be detected, it is determined that the next detection point is not within the line of sight of the current detection point; otherwise, it is determined that the next detection point is within the line of sight of the current detection point.

8. The three-dimensional object surface defect detection method based on multi-robot arm cooperation according to claim 1, characterized in that Step S4 specifically includes the following steps: Step S41: Convert the vertices of the CAD model of the object to be detected from the base coordinate system to the camera coordinate system. After removing the invisible triangular facets, map the three-dimensional space points to the two-dimensional image plane using the pinhole camera principle. Then, perform rasterization processing on the visible triangular facets using the scan-line algorithm to generate a reference image of the object to be detected. Step S42: Establish a light compensation model to eliminate the difference between the actual lighting conditions and the theoretical lighting model, and perform light compensation on the acquired image to obtain a corrected image. The light compensation model is expressed as: Wherein, and represent the image pixel coordinates; represents the light compensation coefficient; represents the gray value of the actual image; represents the expected gray value of the image under the theoretical light model; represents the minimum value, which is used to prevent division-by-zero errors; represents the diffuse reflection coefficient of the material of the object to be detected, and the value range is ; represents the surface normal vector; represents the normalized light source direction vector, which is obtained by mirror sphere calibration; represents the specular reflection coefficient; represents the reflected light direction vector; represents the line-of-sight direction vector; represents the specular highlight exponent, which is used to control the sharpness of specular reflection; Perform Gaussian filtering to eliminate noise and high-frequency fluctuations, obtaining : In the formula, represents the corrected light compensation coefficient; represents the Gaussian kernel, which is used to retain the low-frequency non-uniformity of light; represents the convolution operation; The corrected image is expressed as: Step S43: Calculate the cross-power spectrum of the reference image and the corrected image using the phase correlation method, and obtain the cross-correlation matrix of the reference image and the corrected image through inverse Fourier transform, which is expressed as: In the formula, represents the cross-correlation matrix of the reference image and the corrected image; , represent the spatial frequency coordinate quantities in the frequency domain, represents the corrected image; represents the reference image; represents the two-dimensional Fourier transform, which converts the image to the frequency domain; represents the complex conjugate operation, which is used to calculate the correlation in the frequency domain; represents the inverse Fourier transform; Step S44: Eliminate the positioning error of the robotic arm and adaptively distinguish noise from real defects through sub-pixel image registration and dynamic thresholding. Specifically, it includes the following process: Find the maximum peak coordinates from the cross-correlation matrix , perform quadratic fitting on the peak neighborhood to obtain the sub-pixel offset ; Perform bilinear interpolation on the corrected image to perform sub-pixel translation correction to obtain a corrected image : Step S45: Extract the defect area from the actual image acquired by the robotic arm, and use morphological processing to eliminate noise and connect the real defect areas. Specifically, it includes the following steps: Calculating a corrected image and a difference image with a reference image : Statistical difference graph of the grayscale histogram, and take the average value of the grayscale histogram and standard deviation , set the threshold to , pixels exceeding are regarded as defects; Perform a closing operation on the difference image, dilate and then erode it using a 5x5 circular kernel structuring element to fill small holes; then perform an opening operation, erode and then dilate it using a 5x5 circular kernel structuring element to eliminate isolated noise points. Then, apply the region growing algorithm to accurately extract the continuous defect area. In the morphologically processed image, select the pixels with a gray level difference exceeding 2 times the threshold as the growth starting point, and then check the neighboring pixels. If their gray level difference is less than a certain value, merge them into the growth area; the stopping condition is that the growth reaches the image boundary or 10 consecutive pixels no longer meet the merging condition.

9. The three-dimensional object surface defect detection method based on multi-robot arm cooperation according to claim 8, characterized in that After step S45, the following steps are also included: Step S46: Perform quantitative analysis and classification of the defects through geometric parameter calculation and type discrimination rules. Geometric parameter calculation includes the calculation of defect depth and defect area, where: Defect depth The calculation process is expressed as: In the formula, represents the number of pixels in the defect area; , respectively represent the gray values of the reference image and the actually acquired image after alignment at the pixel point ; is the depth conversion coefficient, which is obtained through calibration; Defect area The calculation process is expressed as: In the formula, represents the image resolution, which is determined by the camera focal length and the working distance; The type of the defect is classified through the discrimination rules: Scratch determination is based on the relationship between the major axis length of the defect area and the minor axis length to determine; specifically: if and the depth gradient direction is consistent with the major axis direction, it is determined as a scratch, is the ratio threshold set manually; The pit determination is based on the defect area and the depth gradient; specifically: if and the depth gradient is radially distributed along the edge, it is determined as a pit, is the area threshold set manually; The crack determination is based on the actual perimeter of the defect area and the perimeter of the equivalent circle The ratio of which is determined by the number of crack branches; specifically: if and the number of branches is greater than or equal to 3, it is determined as a crack is the ratio threshold set manually 10. The three-dimensional object surface defect detection method based on multi-robot arm cooperation according to claim 1, characterized in that, Step S5 specifically includes the following steps: Step S51: Perform coordinate unification processing on the defect data uploaded by each robotic arm, convert the local detection coordinates to the global coordinate system, and generate a global defect point cloud set. The conversion process of the local detection coordinates to the global coordinate system is expressed as: In the formula, represents the coordinates of the defect point in the global coordinate system; represents the coordinates of the defect point in the local detection coordinate system; represents the real-time pose matrix of the end effector, which is calculated from the structural parameters of the robotic arm; represents the pose matrix of the robotic arm base coordinate system relative to the global coordinate system; The global defect point cloud is represented as: , , where represents the total number of local detection points; Step S52: Determine the triangular facet to which each defect point in the global defect point cloud set belongs, and then perform RGB color coding on the triangular facet to which the defect point belongs according to the defect type to achieve defect annotation. The mapping relationship between the defect point and the triangular facet is expressed as: In the formula, represents the triangular patch to which the defect point belongs; represents the operation of projecting the defect point onto the plane of the triangular patch; is the set of triangular patches of the CAD model of the object to be detected; represents the k th triangular patch; The RGB color coding rules are expressed as follows: the color coding of cracks is (255, 0, 0), the color coding of pits is (0, 0, 255), and the color coding of scratches is (0, 255, 0).

Images