An assembly line end point positioning method

Abstract

The application discloses an assembly line end point positioning method, relates to the technical field of machine vision and industrial automation detection, and solves the problem that the parameter of an infinite long straight line is obtained through Hough transform detection, and only approximate end point coordinates can be obtained. The method comprises the following steps: S100, inputting a base plate image, performing contour matching and coarse positioning on the base plate image based on Hu moments; S200, performing preliminary straight line detection on the base plate image after coarse positioning to obtain a plurality of straight line segments; S300, calculating the midpoints and initial directions of the straight line segments, extending along the positive direction and the negative direction respectively from the corresponding midpoints, and collecting line point sets; and S400, performing least square fitting on the collected line point sets to obtain accurate end point coordinates. The method can adapt to illumination changes, accurately position end points, and meets the effect of satisfying the precision requirement of ship small group standing automation assembly.

Description

Technical Field

[0001] This invention relates to the fields of machine vision and industrial automation inspection technology, and in particular to a method for locating the endpoints of an assembly line. Background Technology

[0002] In the assembly process of shipbuilding, the white assembly lines on steel plates are important markers used to indicate the assembly positions of parts. Traditional manual identification methods rely on workers' visual identification of the assembly line positions and endpoints, which has the following problems: low efficiency: manual identification is slow and cannot meet the high-speed requirements of automated production lines; susceptible to subjective factors: different workers have different judgment standards, resulting in unstable assembly accuracy; high labor intensity: long working hours lead to worker fatigue and increase the risk of misjudgment.

[0003] To achieve automated assembly, machine vision technology is needed to automatically identify the positions and endpoints of the assembly line. Current technology primarily uses the Hough Transform for line detection, but this has the following drawbacks: 1. Inaccurate endpoint localization: The Hough transform detects the parameters (ρ, θ) of an infinitely long straight line. Even the probabilistic Hough transform can only obtain approximate endpoint coordinates. In practical applications, the following reasons lead to significant endpoint localization errors: Uneven global brightness: Industrial site lighting conditions are complex, and the brightness of different areas of the steel plate varies greatly. When using a fixed threshold for binarization, the endpoints of lines in dark areas are truncated, while noise is introduced into bright areas. Line brightness variation: The assembly line itself has a gradual change in brightness, with the middle part of the line segment being brighter and the ends gradually darkening, which causes the length of the line segment after binarization to be inconsistent with the actual length. Local contrast variation: Factors such as reflection and shadow on the steel plate surface cause a decrease in local contrast, resulting in blurred edges.

[0004] 2. Sensitive to parameters: The accumulator threshold, minimum line segment length, maximum gap, and other parameters of the Hough transform need to be adjusted according to the specific image. Improper parameter selection can lead to missed or false detections. 3. Noise interference: The surface of the steel plate has interference features such as weld seams, scratches, and oil stains. These interferences are similar to the features of the assembly line and can easily lead to false detection. Summary of the Invention

[0005] The purpose of this invention is to provide an assembly line endpoint positioning method that can adapt to changes in lighting and accurately position endpoints to meet the precision requirements of automated assembly of ship subgroups. This method can accurately position the endpoints of the assembly line under conditions of global brightness changes and local contrast changes, overcome the endpoint positioning deviation caused by the gradual change in brightness of the assembly line, and improve the robustness and real-time performance of the algorithm while ensuring the accuracy of endpoint positioning.

[0006] The above-mentioned technical objective of the present invention is achieved through the following technical solution: An assembly line endpoint positioning method includes the following steps: S100: Input the base plate image, and perform contour matching and coarse localization on the base plate image based on Hu moments; S200: Perform preliminary line detection on the base plate image after coarse positioning to obtain several line segments; S300: Calculate the midpoint and initial direction of each line segment, start from the corresponding midpoint, extend along the positive and negative directions respectively, and collect the line point set; S400: The collected line point set is fitted by least squares to obtain the accurate endpoint coordinates.

[0007] Furthermore, step S100 includes, The template contour is extracted from the theoretical model, and the Hu moment feature vector of the template contour is calculated; the base plate image is input and contour detection is performed to obtain all candidate contours; finally, contour matching based on Hu moment is performed on each candidate contour, and the region of interest is generated according to the theoretical model.

[0008] Furthermore, the contour matching in step S100 includes the following steps: Calculate the Hu moment feature vector of the candidate contour; then perform shape similarity judgment, calculate the Hu moment similarity between the candidate contour and the template contour. If the similarity distance value is less than the threshold, the shape is considered to be successfully matched and proceed to the next step; otherwise, it is judged as a non-base contour and is excluded.

[0009] Furthermore, the coarse positioning in step S100 includes the following steps: First, an image template is generated based on the template outline. Then, multi-angle matching is performed between the image template and the base plate image to obtain the translation and rotation amounts. Next, an affine transformation matrix is ​​constructed based on the translation and rotation amounts. Finally, an affine transformation is performed on the standard assembly line in the theoretical model based on the affine transformation matrix. The obtained theoretical assembly line is then expanded outward by a preset distance to cut out the region of interest.

[0010] Furthermore, the multi-angle matching includes the following steps: Rotate the image template from 0 to 360 degrees, then perform template matching with the base image, calculate the sum of the squared differences of pixels, use the image template as a mask, and only calculate the pixels in the region inside the contour; traverse each angle and image position, and when the sum of the squared differences of pixels inside the contour reaches the minimum value, obtain the optimal matching posture, and output the translation and rotation amount at this time.

[0011] Furthermore, step S200 includes performing adaptive threshold binarization on the coarsely located base plate image to obtain a binary image and performing a probabilistic Hough transform to detect line segments and obtain the preliminary endpoint coordinates of each line segment.

[0012] Furthermore, step 300 includes obtaining the midpoint coordinates and direction vector of the corresponding line segment based on the preliminary endpoint coordinates of each line segment; starting from the midpoint coordinates, extending along the positive and negative directions of the direction vector respectively, with an extension step of S pixels each time, to obtain the prediction point; then extracting the local region around the prediction point and binarizing the local region; scanning within ±R pixels in the vertical direction of the direction vector to extract the coordinates of all white points and calculating their center positions, which are recorded as extension points.

[0013] Furthermore, in step 300, after each extension step, the actual direction is recalculated based on the most recent N extension points.

[0014] Furthermore, in step 300, if no white point is found in a certain step, the process continues until no white point is found after G consecutive steps, at which point the process stops.

[0015] Furthermore, in step S400, after least squares fitting, a fitted straight line is obtained. The first point of the line point set is taken as the starting point and the last point is taken as the ending point. The starting point and the ending point are projected onto the fitted straight line as the precise endpoint coordinates.

[0016] In summary, the present invention has the following beneficial effects: 1) By dynamically adjusting the threshold using local adaptive binarization, focusing the effective area by cropping the ROI, and detecting the core of the line in the vertical direction, the endpoint position of the assembly line can be accurately located under the condition of global brightness change and local contrast change. 2) The design of the bidirectional extension strategy at the midpoint avoids the uncertainty at the end, and the threshold of the low contrast area is automatically reduced. Combined with the gap tolerance mechanism, it overcomes the end point positioning deviation caused by the gradual change of light and dark in the assembly line.

[0017] 3) We propose vertical center detection and least squares fitting to maintain accuracy, Hu moment matching and dynamic direction adjustment to improve robustness, and ROI pruning to reduce computation. This achieves the goal of improving the robustness and real-time performance of the algorithm while ensuring the accuracy of endpoint positioning. In summary, this invention, by employing a combination of coarse positioning of the base plate contour based on Hu moment, a bidirectional extension strategy of the midpoint, local adaptive binarization, and vertical center detection, can achieve sub-pixel-level precise positioning of assembly line endpoints under global brightness and local contrast changes, while meeting industrial real-time requirements. It overcomes the shortcomings of existing Hough transform endpoint positioning, sensitivity to illumination and line brightness gradients, weak noise resistance, and insufficient robustness, effectively solving the core technical challenge of automated positioning in ship assembly. Attached Figure Description

[0018] Figure 1 This is an overall flowchart of an assembly line endpoint positioning method according to the present invention; Figure 2 This is a flowchart of step S100 in an assembly line endpoint positioning method of the present invention; Figure 3 This is the outline diagram of the theoretical model of the foundation plate of a certain group; Figure 4 This is a binarized image of the base plate of a certain group; Figure 5 The detection result of Hough transform line segment in the assembly line endpoint positioning method of the present invention (partial area); Figure 6 A flowchart of precise endpoint positioning via bidirectional midpoint extension in an assembly line endpoint positioning method of the present invention; Figure 7 The assembly line endpoint positioning result in an assembly line endpoint positioning method of the present invention. Detailed Implementation

[0019] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. These embodiments do not constitute a limitation of the present invention. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of this application.

[0020] An assembly line endpoint positioning method is used in the ship assembly process to automatically identify the endpoints of the white assembly line on the steel plate base plate, and it can adapt to changes in lighting and line brightness. Figure 1 As shown, it includes the following steps: S100: Input the base plate image, and perform contour matching and coarse localization on the base plate image based on Hu moments; Specifically, such as Figure 2 As shown, step S100 includes, S110: Extract template contours from theoretical models, including extracting the standard outer contour geometric information of the base plate from theoretical design models (such as DXF files or CAD drawings), calculating the Hu moment feature vector of the template contour, and obtaining the position information of the standard assembly line in the theoretical model. S120: Input the base plate image and perform contour detection, including based on 2D images or depth images, to find the contour edges of image parts and obtain all candidate contours.

[0021] S130: Hu-moment-based contour matching, including performing the following matching steps for each candidate contour; S131: Calculate the Hu moment feature vector H_candidate = [h1, h2, h3, h4, h5, h6, h7] of the candidate contour; S132: Shape similarity determination, including, First, calculate the Hu moment similarity between the candidate contour and the template contour. The similarity calculation rules are as follows: Given two normalized Hu moment vectors:

[0022] Define threshold ε=10 -5 The corrected moment value is: ; Constructing the scaling factor:

[0023] Adaptive weighting, weight allocation strategy:

[0024] The final similarity distance metric (similarity distance value):

[0025] The scaling factor α = 100.

[0026] If the similarity distance value is less than the threshold T_UpBound (range 50-100, preferred value 50), the shape is considered to be successfully matched and proceed to the next step; otherwise, it is determined to be a non-base contour and is excluded.

[0027] S133: Generating image templates based on template outlines, including, The template outline (theoretical outline) is densely sampled and scaled and rounded according to the resolution of the actual base plate image to obtain the outline point set in the image coordinate system. The outline matching template is generated in the form of polygon filling (white is filled inside the outline and black is filled outside the outline), which is the image template. S134: Translation and rotation amounts are obtained based on template multi-angle matching, including, The theoretical image template is rotated from 0 to 360 degrees, and then multi-angle matching of the template is performed with the binarized base image. The sum of the squared differences of the pixels is calculated. The theoretical image template is used as a mask, and only the pixels in the inner region of the contour are calculated, so that the calculation results of different angles remain comparable. By iterating through various angles and image positions, the best matching pose is obtained, which is when the sum of the squared differences of pixels within the contour reaches its minimum value. Then, the best matching angle (rotation amount) and translation amounts (translation_x, translation_y) are output.

[0028] To accelerate angle matching, the 0-360 degree range can initially be divided into larger step intervals. If the best matching angle is best_angle, more refined matching can be performed near best_angle using smaller angle step intervals.

[0029] S135: Construct the affine transformation matrix based on the translation and the best matching angle:

[0030] Where θ is the best_angle, the optimal matching angle. S140: Generate the region of interest (ROI) based on the theoretical model, including performing an affine transformation on the standard assembly line in the theoretical model according to the transformation matrix, mapping it to the actual image coordinate system, expanding the theoretical assembly line outward by a preset distance, and cropping out the region (ROI) where the assembly line may exist, thereby reducing the computational load of subsequent processing and eliminating noise interference from irrelevant regions.

[0031] S200: Perform preliminary straight line detection on the coarsely located base plate image to obtain several straight line segments; specifically including... S210: Image preprocessing, including grayscale conversion and adaptive threshold binarization of the ROI image (region of interest), and extraction of the preliminary outline of white lines; S220: Probabilistic Hough Transform, including performing a probabilistic Hough transform on the binary image obtained in step S210, detecting line segments, and obtaining the preliminary endpoint coordinates (x1, y1) and (x2, y2) of each line segment.

[0032] S300: Precise endpoint positioning via bidirectional midpoint extension. First, calculate the midpoint and initial direction of each line segment. Then, starting from the corresponding midpoint, extend along both the positive and negative directions to collect a set of line points. Specifically, this includes... S310: Calculate the midpoint and direction vector of each line segment based on the initial endpoint coordinates, including, Midpoint coordinates: (mid_x, mid_y) = ((x1+x2) / 2, (y1+y2) / 2); Direction vector: (dx, dy) = (x2-x1, y2-y1); Unit direction vector: (ux, uy) = (dx / L, dy / L), where L is the length of the line segment; S320: Forward extension, including starting from the midpoint and extending gradually along the positive direction (ux, uy), with each extension step being S pixels (S ranges from 1 to 10 pixels, with a preferred value of 5 pixels), to obtain the predicted point: S321: Local adaptive binarization, including, At each step, a local region is extracted around the prediction point (pred_x, pred_y) (window size is W×W pixels, where W ranges from 20 to 70 pixels, with 50 pixels being the preferred value). Then, the maximum gray value of the local region is calculated: maxV = max(local_region), and the median gray value of the local region is calculated: medianV = median(local_region). The threshold is dynamically calculated based on the local contrast. In this embodiment, if maxV - medianV > T1, then the threshold local_threshold = (maxV + medianV + T2) / 2; otherwise, the threshold local_threshold = maxV. Where T1 is the contrast threshold (with a value range of 30-80, preferably 50) and T2 is the bias constant (with a value range of 10-30, preferably 20). Finally, the local area is binarized: if the pixel gray value is greater than local_threshold, it is determined to be a white point (255); otherwise, it is a black point (0).

[0033] S322: Vertical center detection, which scans in a direction perpendicular to the line to find the center position of the white point: including... First, calculate the vertical vector: (perp_x, perp_y) = (-uy, ux). Then, scan within a vertical range of ±R pixels (R ranges from 2 to 10 pixels, with 6 pixels being the preferred value) to extract the coordinates of all white points. If a white point exists, calculate its center position: (center_x, center_y) = (mean(white_points_x), mean(white_points_y)). Record the center position as the extension point and update the current position; S323: Dynamic direction adjustment. After each extension step, the actual direction is recalculated based on the nearest N points (N ranges from 2 to 5, with a preferred value of 3), including... Calculate the direction vector of the nearest N points: (dx_temp, dy_temp) = (x_N-x_1, y_N-y_1), and then obtain a new unit direction vector by normalization. In order to ensure that the direction is basically consistent with the initial direction (to prevent reverse), if the dot product is less than 0, it is inverted. S24: Gap tolerance mechanism, including: if no white point is found in a certain step, increase the gap count gap_count and continue to extend; if no white point is found for G consecutive times (G ranges from 2 to 5, with 3 being the preferred value), stop extending. The above gap tolerance mechanism allows for breaks or dark lines in the lines, and avoids the endpoints being cut off prematurely. S330: Negative extension, which includes starting from the midpoint and performing the same algorithm as the positive extension along the negative direction (-ux, -uy) to collect all points of the reverse extension; S340: Merge bidirectional point sets, including merging the reverse extension point set (in reverse order) and the forward extension point set, removing midpoint duplication, and obtaining a complete line point set.

[0034] S400: The collected set of line points is fitted using least squares to obtain accurate endpoint coordinates, including... S410: Collect the coordinates of all extended points, all_points = [(x1, y1), (x2, y2), ..., (x... n ,y n )]; S420: Perform least squares fitting and use the least squares method to calculate the equation of the straight line ax+by+c=0 (fitted straight line). S430: Endpoint projection, including taking the first point (all_points[0]) of the point set as the starting point and the last point (all_points[-1]) as the ending point; projecting the starting point and the ending point onto the fitted line to obtain the accurate endpoint coordinates; S500 outputs the results, including the geometric parameters of the assembly line, such as the precise start point coordinates (start_point), the precise end point coordinates (end_point), the line segment length (pixels), and the line segment angle (degrees).

[0035] In this embodiment, the above positioning method is applied to the shipbuilding group assembly station, with steel plate size of 1270mm×1960mm, assembly line width of 1-2mm, and length of 570-920mm. The hardware configuration includes an industrial camera (25 megapixels, 5120×5120 resolution), a line scan camera (Gocator2690, 0.3mm resolution setting), a light source (LED strip light source, diffuse illumination), and a computer (Intel i7-10700 CPU, 16GB RAM); the software environment is based on Python 3.8, OpenCV 4.5, and NumPy 1.19. The configuration includes parameter settings, such as Hu moment similarity threshold T_UpBound = 50, local window size W = 50 pixels, extension step S = 5 pixels, vertical scan range R = 9 pixels, number of orientation adjustment points N = 3, and maximum gap G = 3. This embodiment specifically includes the following steps: The first step is to perform base plate contour matching and coarse positioning based on Hu's moment, such as... Figure 2 As shown; (1) Extract the theoretical model contour template. Extract the standard outer contour of the base plate from the DXF file, such as... Figure 3 As shown, the Hu moment eigenvector H_template = [0.1905, 0.004773, 0.001434, 1.9857E-05, -1.3337E-09, -8.619E-07, -3.0809E-09].

[0036] (2) Input image contour detection; a depth image of the substrate is obtained by scanning with a line scan camera, and the height is used as a threshold to obtain a binarized image of the substrate, such as... Figure 4 As shown, OpenCV was used for contour detection, resulting in 631 contours. Then, area filtering was used to obtain one contour (in this example, only one motherboard component was placed).

[0037] (3) Hu moment matching and angle correction; Calculate the Hu moment feature vector H_candidate = [0.1907, 0.004512, 0.001762, 1.9427E-05, -1.2337E-08, -6.349E-07, -7.0239E-08].

[0038] Calculate similarity: Substitute into the similarity distance calculation formula, and the matching distance between the candidate contour and the standard contour is 35, which is less than the set matching threshold T_UpBound, indicating that the candidate contour and the target contour are matched.

[0039] A matching template is generated based on the template outline, with the inside of the outline filled in white and the outside filled in black. This template is then matched against the binarized image obtained from the base plate line scan. Using OpenCV's MatchTemplate function, matching is performed from different angles, and the optimal matching angle is found to be 93°, with a translation amount of X=270mm and a translation amount of Y=300mm.

[0040] (4) Generate ROI: Based on the translation vector (270, 300) and rotation angle of 93 degrees, construct an affine transformation matrix to map the standard assembly line in the theoretical model to the actual image. Generate a 5120×5120 ROI image with the standard assembly line extended 1cm outward as the boundary (the 2D camera is also calibrated at N points, and the start and end points of the assembly line in the image can be calculated through the calibration matrix), and apply it to the 2D image captured by the base plate.

[0041] The second step, preliminary line detection: Adaptive threshold binarization and probabilistic Hough transform are performed on the image after applying the ROI, detecting 39 line segments (including step S2.1 image preprocessing; S2.2 probabilistic Hough transform), as shown below. Figure 5 As shown.

[0042] The third step is to precisely locate the endpoints by extending the midpoint bidirectionally, such as... Figure 6 As shown, including, (1) Calculate the midpoint and initial direction; Taking line segment L5 as an example: Calculate the midpoint: (mid_x, mid_y) = (2375, 1164), calculate the direction vector: (dx, dy) = (374, 158), and the unit direction vector (ux, uy) = (0.921, 0.389). (2) Extending in the positive direction (along the positive direction); Step 1: pred_x = 2380, pred_y = 1166, use local adaptive binarization to find the center point (2380, 1165); Step 2: with pred_x = 2384 and pred_y = 1168, find the center point (2385, 1169); ...(Total 150 steps) Finally, it reaches (3450, 1617); (3) Negative extension (along the negative direction): Step 1: with pred_x = 2370 and pred_y = 1162, find the center point (2370, 1162); Step 2: with pred_x = 2366 and pred_y = 1160, find the center point (2365, 1160). ...(A total of 93 steps); • Finally arrived (1948, 987).

[0043] (4) Merge bidirectional point sets.

[0044] The fourth step is to fit the exact endpoints using least squares. This involves collecting all the extension points, fitting the line using the least squares method, and obtaining the fitting result as y = 0.434x + 134.56. The line is then projected to obtain the starting point projection (1951, 980) and the ending point projection (3445, 1628).

[0045] Step 5: Output the results. Output the precise parameters of assembly line 1: start point: (1951, 980), end point: (3445, 1628), length: 1628 pixels, angle: 23.45 degrees. Similarly, process the remaining line segments, remove duplicate segments, and obtain the final assembly line result. Figure 7 ).

[0046] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within the scope of its essence and protection. Such modifications or equivalent substitutions should also be considered to fall within the protection scope of the present invention.

Claims

1. A method for locating the end point of an assembly line, characterized in that: Includes the following steps, S100: Input the base plate image, and perform contour matching and coarse localization on the base plate image based on Hu moments; S200: Perform preliminary line detection on the base plate image after coarse positioning to obtain several line segments; S300: Calculate the midpoint and initial direction of each line segment, start from the corresponding midpoint, extend along the positive and negative directions respectively, and collect the line point set; S400: The collected line point set is fitted by least squares to obtain the accurate endpoint coordinates.

2. The assembly line endpoint positioning method according to claim 1, characterized in that: Step S100 includes, The template contour is extracted from the theoretical model, and the Hu moment feature vector of the template contour is calculated; the base plate image is input and contour detection is performed to obtain all candidate contours; finally, contour matching based on Hu moment is performed on each candidate contour, and the region of interest is generated according to the theoretical model.

3. The assembly line endpoint positioning method according to claim 2, characterized in that: Contour matching in step S100 includes the following steps: Calculate the Hu moment feature vector of the candidate contour; then perform shape similarity judgment, calculate the Hu moment similarity between the candidate contour and the template contour. If the similarity distance value is less than the threshold, the shape is considered to be successfully matched and proceed to the next step; otherwise, it is judged as a non-base contour and is excluded.

4. The assembly line endpoint positioning method according to claim 1 or 3, characterized in that: The coarse positioning in step S100 includes the following steps: First, an image template is generated based on the template outline. Then, multi-angle matching is performed between the image template and the base plate image to obtain the translation and rotation amounts. Next, an affine transformation matrix is ​​constructed based on the translation and rotation amounts. Finally, an affine transformation is performed on the standard assembly line in the theoretical model based on the affine transformation matrix. The obtained theoretical assembly line is then expanded outward by a preset distance to cut out the region of interest.

5. The assembly line endpoint positioning method according to claim 4, characterized in that: The multi-angle matching includes the following steps. Rotate the image template from 0 to 360 degrees, then perform template matching with the base image, calculate the sum of the squared differences of pixels, use the image template as a mask, and only calculate the pixels in the region inside the contour; traverse each angle and image position, and when the sum of the squared differences of pixels inside the contour reaches the minimum value, obtain the optimal matching posture, and output the translation and rotation amount at this time.

6. The assembly line endpoint positioning method according to claim 1, characterized in that: Step S200 includes performing adaptive threshold binarization on the coarsely located base plate image to obtain a binary image and performing a probabilistic Hough transform to detect line segments and obtain the preliminary endpoint coordinates of each line segment.

7. The assembly line endpoint positioning method according to claim 1, characterized in that: Step 300 includes obtaining the midpoint coordinates and direction vector of the corresponding line segment based on the preliminary endpoint coordinates of each line segment; starting from the midpoint coordinates, extending along the positive and negative directions of the direction vector respectively, with an extension step of S pixels each time, to obtain the prediction point; then extracting the local region around the prediction point and binarizing the local region; scanning within ±R pixels in the vertical direction of the direction vector to extract the coordinates of all white points and calculating their center positions, which are recorded as extension points.

8. The assembly line endpoint positioning method according to claim 7, characterized in that: In step 300, after each extension step, the actual direction is recalculated based on the N most recent extension points.

9. The assembly line endpoint positioning method according to claim 7, characterized in that: In step 300, if no white point is found in a certain step, continue extending until no white point is found after G consecutive steps, then stop extending.

10. The assembly line endpoint positioning method according to claim 1, characterized in that: In step S400, after least squares fitting, a fitted straight line is obtained. The first point of the line point set is taken as the starting point and the last point is taken as the ending point. The starting point and the ending point are projected onto the fitted straight line as the precise endpoint coordinates.

Images