Improved Yolo and adaptive image segmentation preprocessing method for high-reflection metal surface line laser three-dimensional reconstruction

Abstract

The invention belongs to the technical field of three-dimensional reconstruction, and relates to an improved Yolo and adaptive image segmentation preprocessing method for high-reflection metal surface line laser three-dimensional reconstruction. According to the method, an improved Yolo11 network is selected as a target detection model, part of data of a data set is marked by labmeli software and then is divided into a training set and a verification set, a common convolutional layer in the Yolo11 network is improved into snakelike convolution for training, and after the test on the verification set reaches the standard, a target detection result is obtained. And carrying out preliminary detection on all the line laser reconstruction pictures by using the trained model, comparing and screening frame selection areas obtained after detection, and finally cutting the preprocessed line laser reconstruction pictures to carry out two-dimensional Gaussian filtering, graying and binaryzation to complete preprocessing operation. According to the preprocessing method, the noise of high-reflection metal surface reconstruction is greatly reduced, the reconstruction precision is improved, and the reconstruction speed can be improved to a certain extent.

Description

Technical Field

[0001] The present invention belongs to the field of optical three-dimensional image reconstruction; more specifically, it relates to a method for obtaining light stripe areas that undergo multiple reflections on highly reflective metal-like surfaces in combination with Yolo11; the method can be used to locate the position of the light stripe in the image, and then extract the center of the light stripe to obtain three-dimensional reconstruction information of the object surface. Background Art

[0002] In recent years, with breakthroughs in optical sensing, image processing and artificial intelligence technologies, non-contact three-dimensional reconstruction technology based on structured light, laser scanning, stereo vision and other methods has developed rapidly and is widely used in industrial inspection, reverse engineering, intelligent manufacturing, virtual reality and other fields.

[0003] In the field of non-contact 3D reconstruction, line laser-based 3D reconstruction technology is widely used. Its core principle is laser triangulation. By projecting a laser beam onto the surface of an object, the camera is used to capture the deformation image of the laser light strip on the surface of the object, and the calibration parameters are combined to solve the 3D coordinates of the object surface.

[0004] Light stripe center extraction technology is a key research topic in line laser 3D reconstruction. Its core objective is to locate the light stripe and extract its image coordinates. However, the high reflectivity of metal surfaces results in a superposition of specular and diffuse reflections after line laser projection, creating multiple light stripes in the image. Traditional methods for extracting light stripe centers fail to determine the actual projected light stripe position, leading to incorrect calculation of the light stripe center. Summary of the Invention

[0005] To address the above problems, the present invention proposes a preprocessing method combined with the Yolo11 network. By pre-selecting the suspected light strip position, and then locating the area where the real light strip frame is located through a clustering algorithm, the reflected light strip frame is deleted, thereby providing highly reliable input data for subsequent three-dimensional reconstruction. This method improves the reconstruction accuracy and robustness of reflective metal surfaces while ensuring non-contact and high efficiency.

[0006] The technical solution of the present invention:

[0007] An improved Yolo and adaptive image segmentation preprocessing method for line laser 3D reconstruction of highly reflective metal surfaces is proposed. The steps are as follows:

[0008] (1) Collect the multiple reflection image data reconstructed from the highly reflective metal surface, and divide the original reconstructed image data into two parts in proportion, named as training set and validation set respectively.

[0009] (2) Mark the positions of the light stripes in the reconstructed images of the training set.

[0010] (3) The Yolo11 network is used as the deep learning target detection model for light stripes, and the C3K2 convolution layer in Yolo11 is improved to a dynamic snake convolution;

[0011] Furthermore, the dynamic snake convolution kernel is formula (1) (2):

[0012] The change in the x-axis direction is:

[0013]

[0014] The change in the y-axis direction is:

[0015]

[0016] Among them, K is the convolution kernel, K i±c , K j±c is the specific position of each grid in K, c represents the distance from the center grid, Δ={δ|δ∈[-1,1]} is the offset, (x i ,y i )、(x i+c ,y i+c) 、(x i-c ,y i-c ) represent the x and y coordinates of the center point of the convolution kernel and the x and y coordinates of the right and left sides of the center point with a horizontal distance c from the center point, respectively. j ,y j )、(x j+c ,y j+c) 、(x j-c ,y j-c ) represent the x and y coordinates of the center point of the convolution kernel and the x and y coordinates of the upper and lower sides of the center point with a horizontal distance c from the center point.

[0017] (4) Perform deep learning target detection model training, and verify the trained deep learning target detection model using the validation set obtained in step (1). When the mAP tested on the validation set meets the training requirements, the deep learning target detection model is exported, otherwise the parameters are adjusted to continue training.

[0018] Among them, target detection refers to finding all targets of interest in the image and determining their categories and locations.

[0019] (5) Collect a set of line laser 3D reconstruction images and use the deep learning target detection model that meets the requirements of step (4) to detect and annotate them, and obtain the original rectangular frame as the preliminary determined light strip position information.

[0020] (6) The rectangular boxes obtained in step (5) are divided into three different types of categories: long strip, flat, and square according to the aspect ratio and th1 and th2 as thresholds. The long strip is set to a high level, the square is set to a medium level, and the flat is set to a low level; and the coordinates of the center point of each rectangular box are calculated.

[0021] Furthermore, the center coordinates of each rectangular box (x c ,y c ) is calculated as follows:

[0022]

[0023] Where (x c ,y c ) is the coordinate of the center point of the rectangular box, (x, y) is the coordinate of the upper left corner of the rectangular box, width is the width of the rectangular box, height is the length of the rectangular box, th1 is the low threshold of the aspect ratio, and th2 is the high threshold of the aspect ratio.

[0024] (7) The three categories of rectangular boxes divided in step (6) are divided according to the x of the center points of all rectangular boxes in each category. c The coordinates are used to obtain the position number; the decision-making mechanism for position numbering is as follows:

[0025] (7.1) First, initialize all center points to be marked as “unvisited”, traverse each unvisited center point p in the center point set D, mark the traversed center point p as visited, and obtain the set of center points Eps(p) in the ε neighborhood of p. If the number of center points in Eps(p) is less than Minpts, mark p as a noise point and process the next center point. Otherwise, mark p as a core point and create a new cluster C. Add p to C and add all center points in Eps(p) to a queue L.

[0026] (7.2) Then traverse each center point q in the queue L for judgment. If q has not been visited, mark q as visited and obtain the set of center points Eps(q) in the ε neighborhood of q. Determine the number of center points in Eps(q). If it is less than Minpts, mark q as a boundary point. Otherwise, mark q as a core point and add the points in Eps(q) to the queue L. After judging whether it has been visited, determine whether q has been assigned to any cluster. If it has not been assigned to any cluster, add q to the current cluster C.

[0027] (7.3) When all the center points are marked as visited, the cluster set {C1, C2, ..., C k}, and set the position number of each rectangular box to 1, 2, ..., k according to its cluster set number.

[0028] Suppose the clustering sets of the long, flat, and square rectangular boxes are Among them, the position numbers of the long rectangular boxes are 1, 2, ..., k1 according to their cluster set numbers, the position numbers of the flat rectangular boxes are 1, 2, ..., k2 according to their cluster set numbers, and the position numbers of the square rectangular boxes are 1, 2, ..., k3 according to their cluster set numbers.

[0029] (8) Calculate the intersection over union (IoU) of adjacent rectangular boxes in each image from top to bottom in the y direction:

[0030] If max(y 1b ,y 2b ) <max(y 1a ,y 2a ) then set its IoU = 1, otherwise calculate according to formula (4):

[0031]

[0032] where y 1a ,y 2a is the coordinate of the upper rectangular box a, y 1b ,y 2b is the coordinate of the lower rectangular box b.

[0033] (9) Determine the mutual positional relationship of adjacent rectangular frames according to the intersection-and-union ratio calculated in step (8). If the intersection-and-union ratio is equal to 1, the positional relationship is completely contained. If the intersection-and-union ratio is greater than a threshold value T, the positional relationship is a large-scale overlap. If the intersection-and-union ratio is greater than 0 and less than or equal to the threshold value T, the positional relationship is a small-scale overlap. If the intersection-and-union ratio is equal to 0, the positional relationship is non-overlap.

[0034] (10) According to the positional relationship between adjacent rectangular frames in step (9), the rectangular frames are corrected as follows:

[0035] (10.1) First, judge the type category of the rectangular frame obtained in step (6) and determine the rectangular frame with a relatively low type category among the adjacent rectangular frames as the rectangular frame to be processed; if the adjacent rectangular frames are in the same type category, judge again according to the position number in step (7), and determine the rectangular frame with a high position number as the rectangular frame to be processed in the order of position numbers from low to high; if the adjacent rectangular frames are in the same type category and have the same position number, then according to the center point y of the two rectangular frames, c Coordinates are judged, and y c The rectangular frame with the larger coordinates is determined as the rectangular frame to be processed.

[0036] (10.2) According to the positional relationship of adjacent rectangular frames in step (9),

[0037] If the positional relationship is complete inclusion or large-scale overlap, the rectangular box to be processed is directly deleted.

[0038] If the position relationship is a small range overlap, if the rectangle to be processed is the upper rectangle a, then max(y 1a ,y 2a ) is changed to min(y 1b ,y 2b ), if the rectangle to be processed is the lower rectangle b, then min(y 1b ,y 2b ) is changed to max(y 1a ,y 2a ).

[0039] If the position relationship is non-overlapping, the original rectangular frame position is not modified.

[0040] (11) The original captured image is cropped using the final rectangular frame coordinates obtained in step (10).

[0041] (12) Perform two-dimensional Gaussian filtering, grayscale conversion, and binarization on the cropped image to complete image preprocessing.

[0042] The two-dimensional Gaussian function is formula (5)

[0043]

[0044] Where σ is the standard deviation, which controls the degree of smoothing, and (x, y) is the coordinate of the pixel position relative to the center of the kernel.

[0045] The grayscale function is formula (6)

[0046] I(x,y)=0.299×R+0.587×G+0.114×B Formula (6)

[0047] Where B is the blue channel data of the image, G is the green channel data of the image, and R is the red channel data of the image.

[0048] The binarization function is formula (7)

[0049]

[0050] Where I(x,y) is the grayscale value of the input image, and T is the set threshold.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] By combining the Yolo11 network with a preprocessing method for reconstructing highly reflective metal surfaces, the accuracy and robustness of the three-dimensional reconstruction system in dealing with high reflectivity and multiple reflection phenomena on metal surfaces are significantly improved. Compared with traditional methods, this method can effectively avoid errors in determining the position of light strips due to the overlap of mirror reflection and diffuse reflection. By using a deep learning model combined with a clustering algorithm, not only can the light strip area be accurately located, but also misdetected reflected light strips can be eliminated, thereby providing highly reliable input data for subsequent three-dimensional reconstruction. In addition, while ensuring non-contact and high efficiency, the present invention improves the reconstruction accuracy, especially in complex and highly reflective environments, and demonstrates stronger adaptability, providing strong technical support for application fields such as industrial inspection and reverse engineering. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is the overall flow chart of the pretreatment method;

[0054] Figure 2 Reconstruct an image for the original line structured light;

[0055] Figure 3 This is the result of Yolo11 detecting multiple reflections and reconstructing the image;

[0056] Figure 4 Get the flow chart for the rectangular box position number;

[0057] Figure 5 Delete the flowchart for the rectangular box;

[0058] Figure 6 This is the position diagram of the rectangular frame after deletion;

[0059] Figure 7 This is the result image after clipping according to the coordinates of the rectangular frame;

[0060] Figure 8 This is the image after Gaussian blur, grayscale and binarization. DETAILED DESCRIPTION

[0061] In order to more clearly describe the operation steps, technical solutions and advantages of the present invention, the following will take aluminum parts commonly found in industrial products as an example, combined with the accompanying drawings of the present invention, to fully describe the technical steps. The specific implementation plan is as follows: Figure 1 shown.

[0062] (1) Collect 1000 1600×1200 images of highly reflective metal surface reconstruction data, such as Figure 2 As shown in Figure 1, the original reconstructed image data is divided into two parts at a ratio of 4:1, named as training set and validation set respectively.

[0063] (2) Use labelimg software to mark the positions of light stripes in the reconstructed image of the training set. Since only light stripes are detected, only one category, line, is marked during labeling. The label format is saved in txt format.

[0064] (3) Improve the C3K2 convolution layer in Yolo11 to dynamic snake convolution, dynamic snake convolution kernel K i±c , K j±c :

[0065] The change in the x-axis direction is:

[0066]

[0067] The change in the y-axis direction is:

[0068]

[0069] (4) Conduct deep learning target detection model training, and verify the trained deep learning model using the divided validation set. When the mAP tested on the validation set is stable and exceeds 80%, terminate the training and export the model. Otherwise, adjust the parameters, such as adjusting the learning rate, adjusting the number of iterations, etc., and then retrain until the requirements are met.

[0070] (5) A set of reconstructed images are tested using the trained model. The test results are as follows: Figure 3 As shown, the coordinates of the rectangular box are saved in txt format.

[0071] (6) Taking th1 = 0.25, th2 = 0.75 as an example, the rectangles obtained in step (5) are divided into three types: long, flat, and square, and the coordinates of the center point of each rectangle (x c ,y c ):

[0072]

[0073] (7) Taking ε = 2, Minpts = 10 as an example, the three types of rectangular frames divided in step (6) are divided according to Figure 4 The decision-making mechanism constructed is used to obtain the position number:

[0074] (8) Use the vertical coordinate y of the center point of the rectangular frame obtained in step (6) c , calculate the intersection over union (IoU) of adjacent rectangular boxes in each image from small to large:

[0075] If max(y 1b ,y 2b ) <max(y 1a ,y 2a) then set its IoU = 1, otherwise its IoU:

[0076]

[0077] (9) Using the intersection-over-union ratio of adjacent rectangular frames calculated in step (8), taking the threshold T = 0.5 as an example, determine the overlapping relationship of the rectangular frames.

[0078] (10) According to the positional relationship between the rectangular frames in step (9), the coordinates of the original rectangular frame are Figure 5 Modify the constructed selection mechanism, such as Figure 6 As shown, the final coordinates of the rectangular frame are obtained.

[0079] (11) The original captured image is cropped using the final rectangular frame coordinates obtained in step (10). The result is as follows: Figure 7 As shown, the cropped image is saved as a bmp format picture.

[0080] (12) The cropped image obtained in step (11) is subjected to two-dimensional Gaussian filtering, grayscale conversion, and binarization. Figure 8 This is the result after the final preprocessing is completed.

Claims

1. An improved Yolo and adaptive image segmentation preprocessing method for line laser 3D reconstruction of highly reflective metal surfaces, characterized by: Here are the steps: (1) Collect multiple reflection image data reconstructed from highly reflective metal surfaces and divide the original reconstructed image data into two parts in proportion, named as training set and validation set respectively; (2) Mark the positions of the light stripes in the reconstructed images of the training set; (3) The Yolo11 network is used as the deep learning target detection model for light stripes, and the C3K2 convolution layer in Yolo11 is improved to a dynamic snake convolution; (4) Conducting deep learning target detection model training, verifying the trained deep learning target detection model with the validation set obtained in step (1), and exporting the deep learning target detection model when the mAP tested on the validation set meets the training requirements, otherwise adjusting the parameters to continue training; wherein target detection refers to finding all targets of interest in the image and determining their categories and locations; (5) Collect a set of line laser 3D reconstruction images and use the deep learning target detection model that meets the requirements of step (4) to detect and annotate them, and obtain the original rectangular frame as the preliminary determined light strip position information; (6) The rectangular boxes obtained in step (5) are divided into three different types of categories: long strip, flat, and square according to the aspect ratio with th1 and th2 as thresholds. The long strip is set to a high level, the square is set to a medium level, and the flat is set to a low level; and the coordinates of the center point of each rectangular box (x c ,y c ); (7) The three categories of rectangular boxes divided in step (6) are divided according to the x of the center points of all rectangular boxes in each category. c Coordinates to obtain the position number: (8) Calculate the intersection over union (IoU) of adjacent rectangular boxes in each image from top to bottom in the y direction, i.e., the vertical direction; (9) Determine the mutual positional relationship of adjacent rectangular frames according to the intersection-and-union ratio calculated in step (8). If the intersection-and-union ratio is equal to 1, the positional relationship is completely contained. If the intersection-and-union ratio is greater than a threshold value x, the positional relationship is a large-scale overlap. If the intersection-and-union ratio is greater than 0 and less than or equal to a threshold value T, the positional relationship is a small-scale overlap. If the intersection-and-union ratio is equal to 0, the positional relationship is non-overlap. (10) According to the positional relationship between adjacent rectangular frames in step (9), the rectangular frames are corrected as follows: (10.1) First, judge the type category of the rectangular frame obtained in step (6) and determine the rectangular frame with a relatively low type category among the adjacent rectangular frames as the rectangular frame to be processed; if the adjacent rectangular frames are in the same type category, judge again according to the position number in step (7), and determine the rectangular frame with a high position number as the rectangular frame to be processed in the order of position numbers from low to high; if the adjacent rectangular frames are in the same type category and have the same position number, then according to the center point y of the two rectangular frames, c Coordinates are judged, and y c The rectangular frame with larger coordinates is determined as the rectangular frame to be processed; (10.2) According to the positional relationship of adjacent rectangular frames in step (9), If the position relationship is completely contained or widely overlapped, the rectangle to be processed is directly deleted; If the position relationship is a small range overlap, if the rectangle to be processed is the upper rectangle a, then max(y 1a ,y 2a ) is changed to min(y 1b ,y 2b ), if the rectangle to be processed is the lower rectangle b, then min(y 1b ,y 2b ) is changed to max(y 1a ,y 2a ); If the position relationship is non-overlapping, the original rectangular frame position will not be modified; (11) cropping the original captured image using the final rectangular frame coordinates obtained in step (10); (12) Perform two-dimensional Gaussian filtering, grayscale conversion, and binarization on the cropped image to complete image preprocessing.

2. The improved Yolo and adaptive image segmentation preprocessing method for line laser 3D reconstruction of highly reflective metal surfaces according to claim 1, characterized in that: The dynamic snake-shaped convolution kernel in step (3) is represented by formula (1) (2): The change in the x-axis direction is: The change in the y-axis direction is: Among them, K is the convolution kernel, K i±c , K j±c is the specific position of each grid in K, c represents the distance from the center grid, Δ={δ|δ∈[-1,1]} is the offset, (x i ,y i )、(x i+c ,y i+c) 、(x i-c ,y i-c ) represent the x and y coordinates of the center point of the convolution kernel and the x and y coordinates of the right and left sides of the center point with a horizontal distance c from the center point, respectively. j ,y j )、(x j+c ,y j+c) 、(x j-c ,y j-c ) represent the x and y coordinates of the center point of the convolution kernel and the x and y coordinates of the upper and lower sides of the center point with a horizontal distance c from the center point.

3. The improved Yolo and adaptive image segmentation preprocessing method for line laser 3D reconstruction of highly reflective metal surfaces according to claim 1, characterized in that: The center point coordinates (x c ,y c ) is calculated as follows: Where (x c ,y c ) is the coordinate of the center point of the rectangular box, (x, y) is the coordinate of the upper left corner of the rectangular box, width is the width of the rectangular box, height is the length of the rectangular box, th1 is the low threshold of the aspect ratio, and th2 is the high threshold of the aspect ratio.

4. The improved Yolo and adaptive image segmentation preprocessing method for line laser 3D reconstruction of highly reflective metal surfaces according to claim 1, characterized in that: The decision mechanism for position numbering in step (7) is as follows: (7.1) First, initialize all center points to be marked as "unvisited". Traverse each unvisited center point p in the center point set D, mark the traversed center point p as visited, and obtain the set of center points Eps(p) in the ε neighborhood of p. If the number of center points in Eps(p) is less than Minpts, mark p as a noise point and process the next center point. Otherwise, mark p as a core point and create a new cluster C. Add p to C and add all center points in Eps(p) to a queue L. (7.2) Then traverse each center point q in the queue L and make a judgment. If q has not been visited, mark q as visited and obtain the set of center points in q's ε neighborhood, Eps(q). Determine the number of center points in Eps(q). If it is less than Minpts, mark q as a boundary point. Otherwise, mark q as a core point and add the points in Eps(q) to the queue L. After determining whether it has been visited, determine whether q has been assigned to any cluster. If it has not been assigned to any cluster, add q to the current cluster C. (7.3) When all the center points are marked as visited, the cluster set {C1, C2, ..., C k }, and set the position number of each rectangular box to 1, 2, ..., k according to its cluster set number; Suppose the clustering sets of the long, flat, and square rectangular boxes are Among them, the position numbers of the long rectangular boxes are 1, 2, ..., k1 according to their cluster set numbers, the position numbers of the flat rectangular boxes are 1, 2, ..., k2 according to their cluster set numbers, and the position numbers of the square rectangular boxes are 1, 2, ..., k3 according to their cluster set numbers.

5. The improved Yolo and adaptive image segmentation preprocessing method for line laser 3D reconstruction of highly reflective metal surfaces according to claim 1, characterized in that: Step (8) The method for calculating the intersection over union (IoU) of adjacent rectangular boxes is: If max(y 1b ,y 2b ) <max(y 1a ,y 2a ) then set its IoU = 1, otherwise calculate according to formula (4): where y 1a ,y 2a is the coordinate of the upper rectangular box a, y 1b ,y 2b is the coordinate of the lower rectangular box b.

6. The improved Yolo and adaptive image segmentation preprocessing method for line laser 3D reconstruction of highly reflective metal surfaces according to claim 1, characterized in that: The two-dimensional Gaussian function in step (12) is formula (5) Where σ is the standard deviation, which controls the degree of smoothness, and (x, y) is the coordinate of the pixel position relative to the center of the kernel; The grayscale function is formula (6) I(x,y)=0.299×R+0.587×G+0.114×B Formula (6) In the formula, B is the blue channel data of the image, G is the green channel data of the image, and R is the red channel data of the image; The binarization function is formula (7) Where I(x,y) is the grayscale value of the input image, and T is the set threshold.

Images