A convex rivet spatial positioning method based on template matching and density clustering

Abstract

The present application belongs to the field of machine vision, and particularly relates to a convex rivet spatial positioning method based on template matching and density clustering, which comprises the following steps: 1) a line laser camera is used to scan a structure to be measured embedded with rivets in a one-way motion, and a depth image of the whole structure is generated; 2) a matching template of the rivet area depth image is created, and a template file is saved; 3) a rivet local area is found by using the template file to coarsely position the rivet on a new structure to be detected, and the area is converted into a three-dimensional point cloud model; 4) a density clustering algorithm is used to segment the rivet point cloud, a cluster with the maximum Z mean value is calculated to obtain a rivet end point set, and image coordinates are calculated; and 5) the image coordinates are converted into relative coordinates in a world coordinate system of an executing mechanism. The present application does not need an external light source, does not need to set a segmentation threshold, has strong robustness to environmental light changes, and is not affected by the height and form of the rivet. Through coordinate conversion experiment verification, the positioning relative error is not more than 1%.

Description

Technical Field

[0001] This invention belongs to the field of machine vision, specifically a method for spatial localization of protruding rivets based on template matching and density clustering. Background Technology

[0002] In mechanical assembly, many structural components require the use of numerous rivets or screws for connection and fixation. For sealing, corrosion protection, or fastening purposes, the ends of rivets or screws need to be glued with sealant-coated caps. Manually gluing these caps is labor-intensive and inefficient. When using machines to replace manual labor for this task, the first requirement is the ability to accurately measure the spatial position of the rivets on the structural components. Therefore, researching a robust, efficient, and accurate rivet positioning method can enable machines to replace manual labor in gluing rivet caps, improving both gluing efficiency and quality while saving labor costs.

[0003] Currently, machine vision-based methods for rivet identification, detection, and localization mainly fall into two categories: traditional image processing methods and machine learning methods. Traditional image processing methods primarily include Hough circle detection and edge detection, while machine learning methods mainly utilize deep learning. Traditional image processing methods often require a stable external light source when acquiring two-dimensional images, and specific segmentation thresholds need to be set when processing the target region. Furthermore, detection performance is easily affected by changes in lighting and edge shadows. Deep learning methods require a large number of training samples and a cumbersome training process, and often fail to guarantee good results when the shape of the rivet end edge changes. A fast, efficient, and robust rivet spatial localization method could allow machines to more perfectly replace manual labor in rivet cap bonding. Summary of the Invention

[0004] To address the limitations of existing methods in this application scenario, a spatial localization method for protruding rivets based on template matching and density clustering is proposed. This method uses line laser to acquire rivet images, improves the rivet end region extraction method, and uses a template matching algorithm for coarse localization of the rivet. The matched local rivet regions are then converted into 3D point clouds, and density clustering is used to segment and extract the rivet end regions. In the aforementioned scenario, this method solves the problem of unclear boundaries between the rivet edge and the background affecting the results in a 2D plane, and addresses the issue of requiring a large number of training samples and a cumbersome training process when using deep learning to identify rivets. This method does not require setting a threshold, exhibits robustness and adaptability to changes in lighting and rivet shape, reduces the impact of natural light variations on the results, and meets practical requirements in terms of operating efficiency and measurement accuracy. It can quickly and accurately achieve spatial localization of the ends of protruding rivets on structural components without the need for an external light source.

[0005] The technical solution adopted by the present invention to achieve the above objectives is as follows:

[0006] A spatial localization method for protruding rivets based on template matching and density clustering includes the following specific steps:

[0007] 1) Acquire image data of structural components and use threshold segmentation to remove non-structural component main areas from the image;

[0008] 2) Based on the processed image, create and save multiple rivet template files;

[0009] 3) Perform coarse positioning by matching the new image of the structural component to be tested with the template, crop the local area of ​​each rivet according to the original image size, and convert it into a 3D point cloud model;

[0010] 4) The unsupervised DBSCAN density clustering algorithm is used to segment the rivet point cloud model;

[0011] 5) Based on the segmented rivet point cloud model, extract the rivet end point set, and convert the image coordinates of the end center into actual spatial relative coordinates.

[0012] In step 2), the zero-mean normalized cross-correlation matching template file is created using the create_ncc_model operator in Halcon software, and then saved as a .ncm file using the write_ncc_model operator.

[0013] Step 3) includes the following steps:

[0014] 3.1) Use the rivet template as a sliding window and let it slide continuously on the structural component. Calculate the matching cost coefficient once for each slide, find local areas where the matching cost coefficient is greater than the matching threshold, and use these as rivet areas to complete the coarse positioning of the rivets. At the same time, perform an OR operation during the matching process and retain the images of each area at the original image size.

[0015] 3.2) Linearly transform the pixel values ​​of the depth map of the rivet area and use them as the Z-axis coordinates to convert the depth map into a 3D point cloud model.

[0016] The matching cost coefficient ζ is:

[0017]

[0018] Among them, f k Image representing structural component, f t Represents the rivet template image, μ k and μ t f k with f t The pixel mean; x i and yj ζ represents the image coordinates of the pixels within the window; C is the matching window of size M×N; when ζ = -1, the two vectors are considered negatively correlated; when ζ = 0, the two vectors are considered uncorrelated; when ζ ∈ (0,1), the larger ζ is, the more similar the two vectors are considered.

[0019] The linear transformation of the pixel values ​​in the depth map of the rivet area is specifically as follows:

[0020] z i =α·v i +β

[0021] Where α and β are constant coefficients; v i Z represents the pixel value of a point on the image. i This represents the Z-axis coordinate of a pixel after it has been converted into a point cloud model.

[0022] Step 4) includes the following steps:

[0023] 4.1) Define the overall dataset D = {p1, p2, ..., p} for the rivet point cloud model. m}, and the three-dimensional coordinates of the points in the point cloud model are p i =(x i ,y i ,z i ), x i and y i This represents the coordinates of a point in a point cloud model. For any point p... i Calculate the distance from other points in the computation space to p i Euclidean distance Dist(p) i ,p j ), where j represents any integer from 1 to m except i:

[0024]

[0025] 4.2) Set the neighborhood radius ∈, and set Dist(p) to 1. i ,p j Add points ≤ ∈ to p i The neighborhood point set N ∈ (p i ), and |N ∈ (p i Points whose values ​​are greater than or equal to MinPts are denoted as core objects, where MinPts represents the minimum number of neighboring points that should exist within the radius of the neighborhood if the point is denoted as a core object. Find the set of all core object points, denoted as Q.

[0026] 4.3) p in the neighborhood of the core object j By p i Density direct access, find any f k and fk+1 Density-accessible sequences f1, f2, ..., f n Where k = 1, 2, 3, ... n, f1 = p i f n =p j , sequence f k Density concatenation, connecting sequences f1, f2, ..., f n Let it be denoted as point cluster C i Remove C from dataset D i The point in Q is removed from C. i The intersection of the points is then randomly selected as a seed, and all points that are reachable by density from the seed are then identified as new clusters.

[0027] 4.4) Repeat steps 4.1) to 4.3) until the core object point set Q is empty, and those not assigned to cluster C will be excluded. i The points are treated as noise, labeled as -1 and removed to complete the segmentation of the rivet point cloud.

[0028] Step 5) includes the following steps:

[0029] 5.1) Calculate the mean Z-coordinate Z of each cluster after the rivet point cloud model is segmented. i From sequence Z1...Z i Find the maximum value Z max And obtain its corresponding cluster C max C max This refers to the set of points in the rivet end region to be extracted;

[0030] 5.2) Calculate cluster C max The center coordinates are obtained as image coordinates of the center of the rivet end. Based on the calibrated image coordinate origin, the center coordinates of the rivet end are converted into relative coordinates in actual space.

[0031] The present invention has the following beneficial effects and advantages:

[0032] 1. The coarse positioning of rivet matching in this invention adopts the zero-mean normalized cross-correlation (ZNCC) template matching algorithm, which has strong robustness to changes in ambient light. By using multiple templates for matching, high-accuracy coarse matching of rivet areas can be achieved.

[0033] 2. This invention combines the characteristics of convex rivets and structural components, and uses the unsupervised DBSCAN density clustering algorithm to segment the rivet point cloud and calculate the cluster with the largest Z mean, which can achieve fast and accurate extraction of the rivet end region.

[0034] 3. This invention proposes a novel spatial positioning method for convex rivets based on template matching and density clustering. This method does not require an external light source, does not require setting a segmentation threshold, has strong robustness to changes in ambient lighting, and is unaffected by rivet height and shape. Coordinate transformation experiments have verified that the relative positioning error does not exceed 1%. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating the overall process of the method proposed in this invention.

[0036] Figure 2 This is an operational diagram illustrating the creation of a rectangular area rivet template on a structural component depth map according to the present invention.

[0037] Figure 3 This is a point cloud model diagram converted from the coarse positioning area of ​​the rivet obtained after template matching in this invention.

[0038] Figure 4 This invention uses density clustering segmentation Figure 3 Image of the rivet end region extracted from the point cloud model;

[0039] Figure 5 This is a diagram illustrating the rivet positioning effect during an experiment conducted according to the present invention. Detailed Implementation

[0040] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0041] A spatial localization method for protruding rivets based on template matching and density clustering includes the following specific steps:

[0042] 1) The motion mechanism drives the line laser camera to reciprocate and scan the structural components to collect image data;

[0043] 2) Threshold segmentation removes non-structural main areas from the original image to reduce the amount of computation during matching and speed up the calculation.

[0044] 3) Create and save multiple rivet template files;

[0045] 4) Perform coarse positioning on the new image of the structural component to be tested based on the template in step 3), and crop and retain the local area of ​​each rivet according to the original image size, and convert it into a three-dimensional point cloud model.

[0046] 5) The rivet point cloud model is segmented using the unsupervised DBSCAN density clustering algorithm (Density-Based Spatial Clustering of Applications with Noise);

[0047] 6) Calculate the Z-mean of each cluster, retain the cluster with the largest Z-mean to extract the rivet end point set, calculate the image coordinates of the end center and convert them into actual spatial relative coordinates.

[0048] In steps 1) to 2), the Image Acquisition assistant in Halcon software is used to connect a laser camera via the GenICamTL interface protocol to acquire depth images of the structural components. Based on the area threshold, the main area of ​​the structural components is segmented in the scene depth image.

[0049] Step 3) specifically includes:

[0050] The zero-mean normalized cross-correlation (ZNCC) matching template file was created using the create_ncc_model operator in Halcon software, and then saved as a .ncm file using the write_ncc_model operator.

[0051] Step 4) specifically involves:

[0052] 4-1) Use the templates of several rivets to be detected created in step 3) to match on the new structural component depth image. Let the rivet template image slide continuously on the structural component image. Calculate the matching cost coefficient ζ according to formula (1), find the area greater than the matching threshold, and complete the coarse positioning of the rivet matching.

[0053]

[0054] Among them, f k Image representing structural component, f t Represents the rivet template image, μ k and μ t f k with f t The pixel mean; x i and y j ζ represents the image coordinates of the pixels within the window; C is the matching window of size M×N; when ζ = -1, the two vectors are considered negatively correlated; when ζ = 0, the two vectors are considered uncorrelated; when ζ ∈ (0,1), the larger ζ is, the more similar the two vectors are considered.

[0055] During matching, the OR operation is used to improve matching accuracy and error tolerance. The matched local regions are then processed in Halcon software using the reduce_domain operator to retain the original image dimensions for each region.

[0056] 4-2) The pixel values ​​of the depth map of the rivet area obtained in 4-1) are linearly transformed to a reasonable range according to equation (2) and used as the Z-axis coordinates to convert the depth map into a three-dimensional point cloud model, that is:

[0057] z i =α·v i +β#(2)

[0058] Where: α and β are constant coefficients; v i Z represents the pixel value of a point on the image. i This represents the Z-axis coordinate of a pixel after it has been converted into a point cloud model.

[0059] Step 5) specifically refers to:

[0060] 5-1) Define the overall dataset D = {p1, p2, ..., p} for the rivet point cloud model. m}, then define the three-dimensional coordinates of a point in the point cloud model as p i =(x i ,y i ,z i ), here x i and y i This represents the coordinates of a point in the point cloud model, and the x-coordinate in equation (1) of step 4. i and y j The differences they represent. For any point p i Calculate the distance from other points in space to p according to formula (3). i The Euclidean distance, where j represents any integer from 1 to m except i.

[0061]

[0062] 5-2) Set the neighborhood radius ∈, and set Dist(p) i ,p j Add points ≤ ∈ to p i The neighborhood point set N ∈ (p i ). |N ∈ (p i Let the points that are greater than or equal to MinPts be denoted as core objects, and find the set of points of all core objects, denoted as Q.

[0063] 5-3) p in the neighborhood of the core object j By p i Density direct access, find any f k and f k+1 A density-directed sequence f1, f2, ..., fn (k = 1, 2, 3, ..., n) n , where f1 = p i f n =p j , sequence f k Density-connected, this sequence is denoted as point cluster C. i Remove C from set Di The point in Q. Remove the point that intersects with C. i The intersection of the points is then randomly selected as a seed, and all points that are reachable by density from the seed are then identified as new clusters.

[0064] 5-4) Repeat the above steps until the core object point set Q is empty. At this point, most points have been assigned to the corresponding cluster C. i (i = 1, 2, 3... s), where s is the total number of clusters after partitioning. Those clusters not assigned to cluster C... i Points that are not properly segmented are considered noise, are labeled with -1, and then removed. This completes the segmentation of the rivet point cloud.

[0065] Step 6) specifically involves calculating the mean Z-coordinate of each cluster. i From sequence Z1...Z i Find the maximum value Z max And obtain its corresponding cluster C max C max This is the set of points in the rivet end region to be extracted. The center coordinates are calculated to obtain the image coordinates of the rivet end center. Based on the calibrated image coordinate origin, the rivet end center coordinates are converted into relative coordinates in actual space.

[0066] The relationships between the modules of this invention are as follows: Figure 1 As shown, the proposed method framework mainly includes three steps. The first step is data preparation, the second step is rivet matching, and the third step is rivet positioning.

[0067] The present invention specifically includes the following steps:

[0068] 1) The motion mechanism drives the line laser camera to reciprocate and scan the structural components to collect image data;

[0069] 2) Threshold segmentation removes non-structural main areas from the original image to reduce the amount of computation during matching and speed up the calculation.

[0070] 3) Create and save multiple rivet template files;

[0071] 4) Perform coarse positioning on the new image of the structural component to be tested based on the template in step 3), and crop and retain the local area of ​​each rivet according to the original image size, and convert it into a three-dimensional point cloud model.

[0072] 5) The rivet point cloud model is segmented using the unsupervised DBSCAN density clustering algorithm (Density-Based Spatial Clustering of Applications with Noise);

[0073] 6) Calculate the Z-mean of each cluster, retain the cluster with the largest Z-mean to extract the rivet end point set, calculate the image coordinates of the end center and convert them into actual spatial relative coordinates.

[0074] Steps 1) to 3) belong to Figure 1 The specific steps for data preparation are as follows:

[0075] Using the Image Acquisition assistant in Halcon software, a laser camera is connected via the GenICamTL interface protocol. Driven by a motion mechanism, depth images of the structural components are scanned and acquired, preserving the original (partial) image of the main structural component. Figure 2 Manually circle a rectangular template area on it, create a zero-mean normalized cross-correlation (ZNCC) matching template for this area, and save it as a template file with the suffix .ncm. Create as many rivet templates as possible, and improve the matching accuracy and error tolerance by using OR operations during matching.

[0076] Step 4) is Figure 1 For rivet matching, the specific operation is as follows:

[0077] Using the templates created in steps 1) to 3), perform coarse matching on the structural component to be tested. Calculate the matching cost coefficient ζ, which measures the similarity of the matching, according to the following formula. Mark and segment the regions with coefficients greater than the matching threshold using rectangles. In this way, perform coarse matching and positioning of the rivets on the structural component.

[0078]

[0079] The matched local regions are preserved at their original dimensions. The pixel values ​​of the depth map of the rivet region are linearly transformed to a reasonable range using the following formula as the Z-axis coordinate, thus converting the depth map into a 3D point cloud model, where α and β are constant coefficients; v i Z represents the pixel value of a point on the image. i This represents the Z-coordinate of the transformed point cloud model; the transformation effect is as follows. Figure 3 .

[0080] z i =α·v i +β#(2)

[0081] Steps 5) to 6) are Figure 1 The specific steps for the rivet positioning part are as follows:

[0082] The rivet point cloud model obtained in step 4) is segmented using the DBSCAN density clustering algorithm, such as... Figure 4In the overall point cloud model, excluding individual noise clusters, the point set of the rivet end region is the cluster with the largest Z-mean in the point cloud model. Using this, the rivet end region is obtained, and its center is calculated to obtain the rivet image coordinates. Assuming a point in the image has already been marked in the actuator's world coordinate system, the rivet image coordinates are offset to obtain its actual relative coordinates, thus completing its localization. The localization effect is as follows: Figure 5 After multiple experiments, the three-axis coordinate error of the rivet end area center was within ±0.2mm, and the relative error was less than 1%.

[0083] In summary, this invention possesses advantages such as strong robustness, fast computation speed, and accurate positioning in its application scenarios, and can complete the positioning task for rivets of different heights and shapes. The main steps include: proposing a method for spatial positioning of rivets using a combination of template matching and density clustering. Compared to traditional image processing methods, this method does not require setting a segmentation threshold or an external light source, and has strong adaptability and robustness to changes in illumination. Compared to deep learning methods, this method does not require preparing a large number of training samples or performing complex model training; the program runs quickly, has high execution efficiency, and can achieve precise positioning.

[0084] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, extensions, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for spatial positioning of protruding rivets based on template matching and density clustering, characterized in that, The specific steps include the following: 1) Acquire image data of structural components and use threshold segmentation to remove non-structural component main areas from the images; 2) Based on the processed image, create and save multiple rivet template files; 3) Perform coarse positioning by matching the new image of the structural component to be tested with the template, crop the local area of ​​each rivet according to the original image size, and convert it into a 3D point cloud model; 4) The unsupervised DBSCAN density clustering algorithm is used to segment the rivet point cloud model; 5) Based on the segmented rivet point cloud model, extract the rivet end point set, and convert the image coordinates of the end center into actual spatial relative coordinates; Step 5) includes the following steps: 5.1) Calculate the mean Z-coordinate of each cluster after the rivet point cloud model is segmented. Z i From the sequence Z 1 ... Z i Find the maximum value Z max and obtain its corresponding cluster. C max , C max This refers to the set of points in the rivet end region to be extracted; 5.2) Computational Clusters C max The center coordinates are obtained as image coordinates of the center of the rivet end. Based on the calibrated image coordinate origin, the center coordinates of the rivet end are converted into relative coordinates in actual space.

2. The spatial positioning method for protruding rivets based on template matching and density clustering according to claim 1, characterized in that, In step 2), the zero-mean normalized cross-correlation matching template file is created using the create_ncc_model operator in Halcon software, and then saved as a .ncm file using the write_ncc_model operator.

3. The spatial positioning method for protruding rivets based on template matching and density clustering according to claim 1, characterized in that, Step 3) includes the following steps: 3.1) Use the rivet template as a sliding window and let it slide continuously on the structural component. Calculate the matching cost coefficient once for each slide, find local areas where the matching cost coefficient is greater than the matching threshold, and use these as rivet areas to complete the coarse positioning of the rivets. At the same time, perform an OR operation during the matching process, and retain the images of each area at the original image size. 3.2) Linearly transform the pixel values ​​of the depth map of the rivet area and use them as the Z-axis coordinates to convert the depth map into a 3D point cloud model.

4. The spatial positioning method for protruding rivets based on template matching and density clustering according to claim 3, characterized in that, The matching cost coefficient for: ； in, f k Image representing structural components, f t Represents a rivet template image. μ k and μ t They are respectively f k and f t The average pixel value; x i and y j Represents the image coordinates of pixels within the window; C For a matching window of size M×N; when ζ When =-1, the two vectors are considered negatively correlated. ζ When = 0, the two vectors are considered uncorrelated; when ζ When (0,1), ζ The larger the value, the more similar the two vectors are considered to be.

5. The spatial positioning method for protruding rivets based on template matching and density clustering according to claim 3, characterized in that, The linear transformation of the pixel values ​​in the depth map of the rivet area is specifically as follows: ； in, α , β The coefficient is constant. v i Represents the pixel value of a point in the image; Z i This represents the Z-axis coordinate of a pixel after it has been converted into a point cloud model.

6. The spatial positioning method for protruding rivets based on template matching and density clustering according to claim 1, characterized in that, Step 4) includes the following steps: 4.1) Define the overall dataset for the rivet point cloud model. D ={ p 1 , p 2 ,..., p m }, and the three-dimensional coordinates of points in the point cloud model are p i = ( x i , y i , z i ), x i and y i This represents the coordinates of a point in a point cloud model. For any point... p i Compute other points in the computation space to p i European distance ,in, j In addition to representing 1~m i Any integer: ； 4.2) Set the neighborhood radius ,Will Add point p i neighborhood point set ( p i ), will | ( p i )| MinPts The point is designated as the core object, where... MinPts This represents the minimum number of neighboring points that should exist within the radius of a given point if it is considered a core object. The set of all points representing core objects is denoted as […]. Q ; 4.3) Core Objects Neighborhood p j Depend on p i Density direct access, find any f k and f k+1 Density-accessible sequences f 1 , f 2 ,..., f n ,in, k =1,2,3,...n, f 1 = p i , f n = p j ,sequence f k Density-connected, sequence f 1 , f 2 ,..., f n denoted as point cluster C i In the dataset D Remove from middle C i The point in Q Remove the middle C i The intersection of the points is then randomly selected as a seed, and all points that are reachable by density from the seed are then identified as new clusters. 4.4) Repeat steps 4.1) to 4.3) until the core object point set is reached. Q If empty, it will not be assigned to a cluster. C i The points are treated as noise, labeled as -1 and removed to complete the segmentation of the rivet point cloud.

Images