An automatic detection method for weld residual height by integrating point cloud and image segmentation

Abstract

The present invention relates to the field of special equipment residual height detection technology, and discloses an automatic detection method for weld residual height that integrates point cloud and image segmentation. By combining binocular stereo vision, AI analysis and point cloud analysis technology, the method can quickly measure the misaligned residual height and non-misaligned residual height of welds, make up for the shortcomings of traditional detection methods in efficiency and accuracy, and provide a reliable technical means for actual detection; the method comprises the following specific steps: photographing the object to be measured by a binocular structured light camera, obtaining an aligned grayscale image and point cloud image by binocular stereo vision technology; performing target segmentation on the grayscale image, finding the weld in the grayscale image, and segmenting the weld from the background to generate a weld segmentation binary mask image; generating a position area mask according to the weld segmentation binary mask image, and further distinguishing it into an upper parent material area, a weld area and a lower parent material area, generating a regional mask image, and naming the regional mask image box_mask, and using it in conjunction with a point cloud image corresponding to the original grayscale image.

Description

Technical Field

[0001] The present invention relates to the field of special equipment residual height detection technology, and in particular to an automatic detection method for weld residual height by integrating point cloud and image segmentation. Background Art

[0002] Welding technology is widely used in the manufacturing of specialized equipment, such as boilers and amusement rides. The quality of welds in these devices directly impacts their safety and reliability. Weld reinforcement, a key indicator of weld quality, reflects the vertical protrusion of the weld, which in turn affects the structural strength and service life of the equipment.

[0003] Existing weld reinforcement detection methods primarily include manual visual inspection, two-dimensional image processing, contact measurement equipment, laser scanning, and point cloud analysis. Manual inspection relies on the operator's experience, resulting in high subjectivity and low detection efficiency. While two-dimensional image processing methods offer a certain degree of automation, they lack depth information and are unable to accurately reflect the three-dimensional shape of the weld, resulting in limited detection accuracy. Contact measurement equipment such as coordinate measuring machines (CMMs), while highly accurate, are expensive and complex to operate, making them difficult to implement in large-scale production. Laser scanning and point cloud analysis methods can provide relatively comprehensive three-dimensional information, but their expensive equipment, complex data processing, and demanding working environment limit their practical application.

[0004] Therefore, there is an urgent need for an automatic detection method for weld excess height that can integrate point cloud and image segmentation technology to improve the accuracy and efficiency of detection, reduce costs, and adapt to the actual needs of special equipment manufacturing. Summary of the Invention

[0005] The purpose of the present invention is to provide an automatic detection method for weld excess height by integrating point cloud and image segmentation. By combining binocular stereo vision, AI analysis and point cloud analysis technology, this method can quickly measure the misaligned excess height and the correct excess height of the weld, making up for the shortcomings of traditional detection methods in efficiency and accuracy, and providing a reliable technical means for actual detection.

[0006] The present invention is implemented by the following technical solution: a method for automatically detecting weld excess height by integrating point cloud and image segmentation, comprising the following specific steps:

[0007] 1) Use a binocular structured light camera to shoot the object to be measured, and use binocular stereo vision technology to obtain the aligned grayscale image and point cloud image;

[0008] 2) Perform target segmentation on the grayscale image, find the weld in the grayscale image, and segment the weld from the background to obtain a weld segmentation binary mask image;

[0009] 3) Generate a position area mask based on the weld segmentation binary mask map, and further divide it into the upper parent material area, weld area, and lower parent material area. Generate a regional mask map and name it box_mask. It is used in conjunction with the point cloud map corresponding to the original grayscale image.

[0010] 4) Based on the region mask map (box_mask), extract the upper and lower boundary lines of the weld from the minimum closure area to generate the weld boundary mask map (weld upper boundary mask and weld lower boundary mask):

[0011] 5) Map the regional mask map and weld boundary mask map to the point cloud map, and obtain the point cloud data and point cloud boundaries of the sub-regions;

[0012] 6) By integrating RANSAC and the least squares method, a surface fitting method is used to fit the base material point cloud below the weld and obtain the lower surface equation. The X and Y values of the upper boundary point cloud of the weld are then substituted into the equation to calculate the corresponding Z value at the X and Y points. This value is then subtracted from the initial Z value of the upper boundary point cloud of the weld to obtain the height difference of the fitted point cloud. This is used to determine whether there is a misalignment during the welding process. When the mean value of the point cloud height difference is greater than 1 mm, it is determined that there is a misalignment; when the mean value of the point cloud height difference is less than 1 mm, it is determined that there is no misalignment.

[0013] 7) If there is no misalignment, the weld area point cloud is denoised based on density and distance statistics. The point with the highest Z value in the denoised weld area point cloud is then selected as the highest point of the weld. The vertical distance (Z direction) from the highest point of the weld to the upper and lower surface equations obtained by surface fitting is calculated respectively. The maximum value of the two is finally taken as the weld height value.

[0014] 8) If there is a misalignment, the points in the weld upper boundary point cloud and the weld lower boundary point cloud are matched one by one. For each pair of corresponding weld upper boundary point cloud and weld lower boundary point cloud, three sampling interpolation is performed along the slope direction of the line connecting the two points to generate all interpolation point sets; after all interpolation is completed, a surface fitting method that integrates RANSAC and least squares method is used to perform plane fitting on these interpolation point sets to obtain the plane equation;

[0015] 9) After step 8), perform point cloud denoising on the weld area point cloud based on density and distance statistics. Using the plane equation obtained in step 8), calculate the vertical distance from each weld point to the plane fitted in step 8) using the point-to-plane distance formula. Take the maximum distance value as the weld reinforcement value.

[0016] To further better implement the method for automatically detecting weld excess height by integrating point cloud and image segmentation according to the present invention, the following setting is particularly adopted: Step 1) includes the following specific steps:

[0017] 1.1) Obtain a left grayscale image and a right grayscale image through the left and right viewing angles of the binocular structured light camera, respectively. The left grayscale image and the right grayscale image together constitute a grayscale image;

[0018] 1.2) Perform epipolar correction (Rectification) on the left grayscale image and the right grayscale image to ensure that the left grayscale image and the right grayscale image are aligned on the same horizontal line to obtain the aligned grayscale image;

[0019] 1.3) After step 1.2), a stereo matching algorithm (such as Block Matching or Semi-Global Matching) is used to obtain a disparity map.

[0020] 1.4) Combining the disparity map with the camera's intrinsic and extrinsic parameters (including the camera's baseline, focal length, optical center, and other parameters), triangulate each pixel in the left and right grayscale images to its coordinate point in 3D space.

[0021] 1.5) Assign the grayscale value or color information of each pixel to the corresponding 3D point, forming a one-to-one correspondence between "pixel and point cloud", thereby obtaining a point cloud image with the same resolution as the grayscale image.

[0022] In order to further better implement the automatic detection method of weld residual height by integrating point cloud and image segmentation described in the present invention, the following setting method is particularly adopted: in step 2), when performing target segmentation, the target segmentation network V8-seg is used to complete it, and the weld segmentation area and the actual minimum closure box obtained based on the weld segmentation area are obtained.

[0023] To further better implement the method for automatically detecting weld excess height by integrating point cloud and image segmentation according to the present invention, the following setting is particularly adopted: Step 3) includes the following steps:

[0024] 3.1) Use contour extraction methods (such as the minimum bounding rectangle / minimum closure algorithm in OpenCV) to process the weld segmentation binary mask image to obtain the preliminary minimum closure area;

[0025] 3.2) Map the preliminary minimum closure region to an image of the same size as the initial grayscale image to generate a region mask. On this region mask (which can be considered a labeled image of the same size as the original grayscale image), mark the weld area into the middle region, the upper base metal region, and the lower base metal region.

[0026] 3.3) After step 3.2), the generated region mask image is named "box_mask" and used in conjunction with the point cloud image corresponding to the original grayscale image.

[0027] To further better implement the method for automatically detecting weld excess height by integrating point cloud and image segmentation according to the present invention, the following configuration is particularly adopted: Step 3.1) includes the following specific steps:

[0028] 3.1.1) First, obtain the contour of the weld mask and use the minimum closure algorithm to obtain the four corner points of the contour;

[0029] 3.1.2) Select the upper left and upper right corners as the starting and ending points, and interpolate according to the slope between the two points to obtain the upper boundary of the minimum closure;

[0030] 3.1.3) Select the lower left and lower right corners as the starting and ending points, and interpolate according to the slope between the two points to obtain the lower boundary of the minimum closure;

[0031] 3.1.4) Using the upper left corner and lower left corner as the starting and ending points, interpolate according to the slope between the two points to obtain the left boundary of the minimum closure;

[0032] 3.1.5) Using the upper right corner and lower right corner as the starting and ending points, interpolate according to the slope between the two points to obtain the right boundary of the minimum closure;

[0033] 3.1.6) After steps 3.1.1) to 3.1.5), the complete preliminary minimum closure region is finally obtained.

[0034] To further better implement the method for automatically detecting weld excess height by integrating point cloud and image segmentation according to the present invention, the following setting is particularly adopted: Step 4) includes the following specific steps:

[0035] 4.1) In box_mask, use the four endpoints of the minimum closure to select the upper left corner and the upper right corner, interpolate according to the slope between the two points, obtain the upper boundary line, and generate the weld upper boundary mask;

[0036] 4.2) Select the lower left corner point and the lower right corner point, interpolate according to the slope between the two points, obtain the lower boundary line and generate the weld lower boundary mask;

[0037] 4.3) After steps 4.1) and 4.2), a weld boundary mask image is obtained.

[0038] To further better implement the method for automatically detecting weld excess height by integrating point cloud and image segmentation according to the present invention, the following setting is particularly adopted: Step 5) includes the following steps:

[0039] 5.1) Since each pixel in the grayscale image corresponds one-to-one to the point cloud, map the pixel index of the region mask image (box_mask) to the point cloud image with the same index (u, v) so that a corresponding "region label" can be added to each point in the point cloud image; for example:

[0040] Label 0: base material point cloud in the weld area;

[0041] Label 1: point cloud of weld area;

[0042] Label 2: Point cloud of base material in the area below the weld.

[0043] 5.2) Since the boundary mask contains the pixel information of the upper and lower boundary lines of the weld, the coordinate index (u, v) corresponding to each pixel in the weld boundary mask image is mapped to the point cloud image to obtain the weld boundary point cloud in 3D space;

[0044] 5.3) Group the points in the point cloud according to the location of the "region label":

[0045] Base material point cloud on weld: all 3D points under the "region label" of the base material area;

[0046] Weld area point cloud: 3D points under the "Area Label" in the middle area;

[0047] Base material point cloud under the weld: 3D points under the "Area Label" in the base material area;

[0048] Weld upper boundary point cloud: 3D points under the "region label" at the weld upper boundary mask;

[0049] Weld lower boundary point cloud: 3D points under the "Region Label" at the weld lower boundary mask.

[0050] To further better implement the automatic detection method for weld reinforcement by integrating point cloud and image segmentation described in the present invention, the following setting is particularly adopted: in step 7), the vertical distances from the highest point of the weld to the upper and lower surface equations obtained by surface fitting are respectively calculated, and the maximum value of the two is finally taken as the weld reinforcement value. The specific process is as follows:

[0051] 7.1) Perform surface fitting on the base material point cloud below the weld and obtain the lower surface equation. Perform surface fitting on the base material point cloud above the weld and obtain the upper surface equation.

[0052] 7.2) Substitute the X and Y values of the highest point of the weld into the upper surface equation to calculate the corresponding Z value at X and Y. Calculate the difference between the Z value of the highest point of the weld and the calculated Z value of the upper surface, and record this difference as the residual height value.

[0053] 7.3) Substitute the X and Y values of the highest point of the weld into the equation of the lower surface to calculate the corresponding Z value at X and Y. Calculate the difference between the Z value of the highest point of the weld and the calculated Z value of the lower surface, and record the difference as another residual height value;

[0054] 7.4) The maximum of the two reinforcement values is taken as the final reinforcement value of the weld.

[0055] In order to further better realize the automatic detection method of weld excess height by integrating point cloud and image segmentation described in the present invention, the following setting method is particularly adopted: the grayscale image includes a left grayscale image obtained by the left perspective of the binocular structured light camera and a right grayscale image obtained by the right perspective of the binocular structured light camera; the point cloud image can also be obtained by using binocular vision stereo imaging technology, multi-perspective reconstruction, monocular vision stereo imaging technology or NERF technology to obtain the grayscale image, and in the process of reconstructing the grayscale image to the point cloud image, an alignment operation from the 2D image to the 3D point cloud image is also performed.

[0056] The specific steps of obtaining the point cloud map using binocular vision stereo imaging technology are as follows:

[0057] First, speckle structured light is projected onto the left grayscale image and the right grayscale image to obtain the left grayscale image and the right grayscale image with speckle structured light;

[0058] Then, feature points are matched on the two grayscale images, and the point cloud image of the object is obtained through binocular stereo matching technology.

[0059] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0060] In the inspection industry for special equipment such as pressure vessels, boilers, and pipelines, this method can be applied to weld reinforcement measurement and quality assessment. It is suitable for quality control during the manufacturing process and rapid testing during regular inspections, ensuring the safety and reliability of equipment operation. In boiler inspections, this method can be used to measure reinforcement and assess defects on boiler welds and pressure component surfaces, helping to improve the safety and stability of equipment operation.

[0061] The present invention proposes an automatic residual height analysis method based on 3D scanning point cloud, which uses original data for analysis and has more accurate results.

[0062] During the calculation process, the present invention can determine whether there is a misalignment through plane fitting using the point cloud of the initial surface, and automatically calculate the residual height of the misalignment and the residual height of the non-misalignment.

[0063] The present invention combines 2D and 3D technologies to speed up analysis and can be performed on edge deployment boards, making the analysis lightweight.

[0064] The present invention uses binocular structured light stereo imaging technology to form a point cloud with an accuracy of up to 0.01mm. The analysis is performed directly through the scanning surface, which greatly improves the accuracy of the analysis.

[0065] The present invention can perform calculations on all areas of a weld, rather than simply providing values on one area or one line.

[0066] The present invention can distinguish between wrong edges and correct edges by fitting the initial plane, and can construct different underlying algorithms for different situations.

[0067] The present invention adopts the combination of 2D+3D, and the algorithm can be deployed on lightweight edge devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 The binocular structured light camera captures the actual image (grayscale image) of the object being measured.

[0069] Figure 2 It is the point cloud image obtained after the grayscale image is processed.

[0070] Figure 3 This is the result image obtained after grayscale image segmentation (the superposition of the binary mask image and the original image).

[0071] Figure 4 is the region mask image.

[0072] Figure 5 This is the weld boundary mask image.

[0073] Figure 6 It is the point cloud grouping map obtained after mapping the region mask map.

[0074] Figure 7 The upper and lower boundary point cloud grouping images are obtained after mapping the weld boundary mask image.

[0075] Figure 8 This is the extended diagram of the upper and lower parent material point cloud surface fitting.

[0076] Figure 9 This is a good side diagram.

[0077] Figure 10 This is an overview diagram of the calculation of the residual height of the staggered side.

[0078] Figure 11 This is the calculation diagram of the plane coheight of the misaligned boundary point cloud.

[0079] Figure 12 This is a 2D display of the maximum value area of the point cloud coheight. DETAILED DESCRIPTION

[0080] The present invention will be further described in detail below with reference to the examples, but the embodiments of the present invention are not limited thereto.

[0081] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention for which protection is sought, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0082] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature identified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0083] Example 1:

[0084] The present invention designs an automatic weld excess height detection method that integrates point cloud and image segmentation. By combining binocular stereo vision, AI analysis, and point cloud analysis technology, this method can quickly measure the weld excess height of the wrong side and the right side, making up for the shortcomings of traditional detection methods in efficiency and accuracy, and providing a reliable technical means for actual detection. The method includes the following specific steps:

[0085] 1) Use a binocular structured light camera to shoot the object to be measured, and use binocular stereo vision technology to obtain the aligned grayscale image and point cloud image;

[0086] 2) Perform target segmentation on the grayscale image, find the weld in the grayscale image, and segment the weld from the background to obtain a weld segmentation binary mask image;

[0087] 3) Generate a position area mask based on the weld segmentation binary mask map, and further divide it into the upper parent material area, weld area, and lower parent material area. Generate a regional mask map and name it box_mask. It is used in conjunction with the point cloud map corresponding to the original grayscale image.

[0088] 4) Based on the region mask map (box_mask), extract the upper and lower boundary lines of the weld from the minimum closure area to generate the weld boundary mask map (weld upper boundary mask and weld lower boundary mask):

[0089] 5) Map the regional mask map and weld boundary mask map to the point cloud map, and obtain the point cloud data and point cloud boundaries of the sub-regions;

[0090] 6) By integrating RANSAC and the least squares method, a surface fitting method is used to fit the base material point cloud below the weld and obtain the lower surface equation. The X and Y values of the upper boundary point cloud of the weld are then substituted into the equation to calculate the corresponding Z value at the X and Y points. This value is then subtracted from the initial Z value of the upper boundary point cloud of the weld to obtain the height difference of the fitted point cloud. This is used to determine whether there is a misalignment during the welding process. When the mean value of the point cloud height difference is greater than 1 mm, it is determined that there is a misalignment; when the mean value of the point cloud height difference is less than 1 mm, it is determined that there is no misalignment.

[0091] 7) If there is no misalignment, the weld area point cloud is denoised based on density and distance statistics. The point with the highest Z value in the denoised weld area point cloud is then selected as the highest point of the weld. The vertical distance (Z direction) from the highest point of the weld to the upper and lower surface equations obtained by surface fitting is calculated respectively. The maximum value of the two is finally taken as the weld height value.

[0092] 8) If there is a misalignment, the points in the weld upper boundary point cloud and the weld lower boundary point cloud are matched one by one. For each pair of corresponding weld upper boundary point cloud and weld lower boundary point cloud, three sampling interpolation is performed along the slope direction of the line connecting the two points to generate all interpolation point sets; after all interpolation is completed, a surface fitting method that integrates RANSAC and least squares method is used to perform plane fitting on these interpolation point sets to obtain the plane equation;

[0093] 9) After step 8), perform point cloud denoising on the weld area point cloud based on density and distance statistics. Using the plane equation obtained in step 8), calculate the vertical distance from each weld point to the plane fitted in step 8) using the point-to-plane distance formula. Take the maximum distance value as the weld reinforcement value.

[0094] Example 2:

[0095] This embodiment is a further optimization based on the above embodiment. The similarities with the above technical solutions are not repeated here. In order to better implement the automatic detection method of weld excess height by integrating point cloud and image segmentation according to the present invention, the following setting is particularly adopted: Step 1) includes the following specific steps:

[0096] 1.1) Obtain a left grayscale image and a right grayscale image through the left and right viewing angles of the binocular structured light camera, respectively. The left grayscale image and the right grayscale image together constitute a grayscale image;

[0097] 1.2) Perform epipolar correction (Rectification) on the left grayscale image and the right grayscale image to ensure that the left grayscale image and the right grayscale image are aligned on the same horizontal line to obtain the aligned grayscale image;

[0098] 1.3) After step 1.2), a stereo matching algorithm (such as Block Matching or Semi-Global Matching) is used to obtain a disparity map.

[0099] 1.4) Combining the disparity map with the camera's intrinsic and extrinsic parameters (including the camera's baseline, focal length, optical center, and other parameters), triangulate each pixel in the left and right grayscale images to its coordinate point in 3D space.

[0100] 1.5) Assign the grayscale value or color information of each pixel to the corresponding 3D point, forming a one-to-one correspondence between "pixel and point cloud", thereby obtaining a point cloud image with the same resolution as the grayscale image.

[0101] Example 3:

[0102] This embodiment is further optimized based on any of the above embodiments, and the similarities with the above technical solutions are not repeated here. In order to better realize the automatic detection method of weld residual height by integrating point cloud and image segmentation as described in the present invention, the following setting method is particularly adopted: In step 2), when performing target segmentation, the target segmentation network V8-seg is used to complete it, and the weld segmentation area and the actual minimum closure box obtained based on the weld segmentation area are obtained.

[0103] Example 4:

[0104] This embodiment is a further optimization based on any of the above embodiments. The similarities with the above technical solutions are not repeated here. In order to better implement the automatic detection method of weld excess height by integrating point cloud and image segmentation according to the present invention, the following setting is particularly adopted: Step 3) includes the following steps:

[0105] 3.1) Use contour extraction methods (such as the minimum bounding rectangle / minimum closure algorithm in OpenCV) to process the weld segmentation binary mask image to obtain the preliminary minimum closure area;

[0106] 3.2) Map the preliminary minimum closure region to an image of the same size as the initial grayscale image to generate a region mask. On this region mask (which can be considered a labeled image of the same size as the original grayscale image), mark the weld area into the middle region, the upper base metal region, and the lower base metal region.

[0107] 3.3) After step 3.2), the generated region mask image is named "box_mask" and used in conjunction with the point cloud image corresponding to the original grayscale image.

[0108] Example 5:

[0109] This embodiment is a further optimization based on any of the above embodiments. The similarities with the above technical solutions are not repeated here. In order to better implement the automatic detection method of weld excess height by integrating point cloud and image segmentation described in the present invention, the following setting is particularly adopted: Step 3.1) includes the following specific steps:

[0110] 3.1.1) First, obtain the contour of the weld mask and use the minimum closure algorithm to obtain the four corner points of the contour;

[0111] 3.1.2) Select the upper left and upper right corners as the starting and ending points, and interpolate according to the slope between the two points to obtain the upper boundary of the minimum closure;

[0112] 3.1.3) Select the lower left and lower right corners as the starting and ending points, and interpolate according to the slope between the two points to obtain the lower boundary of the minimum closure;

[0113] 3.1.4) Using the upper left corner and lower left corner as the starting and ending points, interpolate according to the slope between the two points to obtain the left boundary of the minimum closure;

[0114] 3.1.5) Using the upper right corner and lower right corner as the starting and ending points, interpolate according to the slope between the two points to obtain the right boundary of the minimum closure;

[0115] 3.1.6) After steps 3.1.1) to 3.1.5), the complete preliminary minimum closure region is finally obtained.

[0116] Example 6:

[0117] This embodiment is a further optimization based on any of the above embodiments. The similarities with the above technical solutions are not repeated here. In order to better implement the automatic detection method of weld excess height by integrating point cloud and image segmentation according to the present invention, the following setting is particularly adopted: Step 4) includes the following specific steps:

[0118] 4.1) In box_mask, use the four endpoints of the minimum closure to select the upper left corner and the upper right corner, interpolate according to the slope between the two points, obtain the upper boundary line, and generate the weld upper boundary mask;

[0119] 4.2) Select the lower left corner point and the lower right corner point, interpolate according to the slope between the two points, obtain the lower boundary line and generate the weld lower boundary mask;

[0120] 4.3) After steps 4.1) and 4.2), a weld boundary mask image is obtained.

[0121] Example 7:

[0122] This embodiment is a further optimization based on any of the above embodiments. The similarities with the above technical solutions are not repeated here. In order to better implement the automatic detection method of weld residual height by integrating point cloud and image segmentation according to the present invention, the following setting is particularly adopted: Step 5) includes the following steps:

[0123] 5.1) Since each pixel in the grayscale image corresponds one-to-one to the point cloud, map the pixel index of the region mask image (box_mask) to the point cloud image with the same index (u, v) so that a corresponding "region label" can be added to each point in the point cloud image; for example:

[0124] Label 0: base material point cloud in the weld area;

[0125] Label 1: point cloud of weld area;

[0126] Label 2: Point cloud of base material in the area below the weld.

[0127] 5.2) Since the boundary mask contains the pixel information of the upper and lower boundary lines of the weld, the coordinate index (u, v) corresponding to each pixel in the weld boundary mask image is mapped to the point cloud image to obtain the weld boundary point cloud in 3D space;

[0128] 5.3) Group the points in the point cloud according to the location of the "region label":

[0129] Base material point cloud on weld: all 3D points under the "region label" of the base material area;

[0130] Weld area point cloud: 3D points under the "Area Label" in the middle area;

[0131] Base material point cloud under the weld: 3D points under the "Area Label" in the base material area;

[0132] Weld upper boundary point cloud: 3D points under the "region label" at the weld upper boundary mask;

[0133] Weld lower boundary point cloud: 3D points under the "Region Label" at the weld lower boundary mask.

[0134] Example 8:

[0135] This embodiment is a further optimization based on any of the above embodiments. The similarities with the above technical solutions are not repeated here. In order to better implement the automatic detection method of weld reinforcement by integrating point cloud and image segmentation described in the present invention, the following setting is particularly adopted: in the step 7), the vertical distances from the highest point of the weld to the upper and lower surface equations obtained by surface fitting are calculated respectively, and the maximum value of the two is finally taken as the weld reinforcement value. The specific process is as follows:

[0136] 7.1) Perform surface fitting on the base material point cloud below the weld and obtain the lower surface equation. Perform surface fitting on the base material point cloud above the weld and obtain the upper surface equation.

[0137] 7.2) Substitute the X and Y values of the highest point of the weld into the upper surface equation to calculate the corresponding Z value at X and Y. Calculate the difference between the Z value of the highest point of the weld and the calculated Z value of the upper surface, and record this difference as the residual height value.

[0138] 7.3) Substitute the X and Y values of the highest point of the weld into the equation of the lower surface to calculate the corresponding Z value at X and Y. Calculate the difference between the Z value of the highest point of the weld and the calculated Z value of the lower surface, and record the difference as another residual height value;

[0139] 7.4) The maximum of the two reinforcement values is taken as the final reinforcement value of the weld.

[0140] Example 9:

[0141] This embodiment is further optimized on the basis of any of the above embodiments, and the similarities with the above technical solutions are not repeated here. In order to better realize the automatic detection method of weld excess height by integrating point cloud and image segmentation as described in the present invention, the following setting method is particularly adopted: the grayscale image includes a left grayscale image obtained by the left perspective of the binocular structured light camera and a right grayscale image obtained by the right perspective of the binocular structured light camera; the point cloud image can also be obtained by using binocular vision stereo imaging technology, multi-perspective reconstruction, monocular vision stereo imaging technology or NERF technology to convert the grayscale image into a point cloud image, and in the process of reconstructing the grayscale image into the point cloud image, an alignment operation from the 2D image to the 3D point cloud image is also performed.

[0142] The specific steps of obtaining the point cloud map using binocular vision stereo imaging technology are as follows:

[0143] First, speckle structured light is projected onto the left grayscale image and the right grayscale image to obtain the left grayscale image and the right grayscale image with speckle structured light;

[0144] Then, feature points are matched on the two grayscale images, and the point cloud image of the object is obtained through binocular stereo matching technology.

[0145] Example 10:

[0146] An automatic detection method for weld reinforcement height by integrating point cloud and image segmentation includes the following steps:

[0147] 1. Use a binocular structured light camera to shoot the object to be measured. The actual image obtained by the binocular structured light camera from the left perspective is as follows: Figure 1 As shown, it is a grayscale image.

[0148] 2. The binocular structured light camera projects speckle structured light on the left grayscale image and the right grayscale image (the left grayscale image and the right grayscale image constitute a grayscale image) obtained by the left lens and the right lens, and obtains a grayscale image with speckle structured light. Feature points are matched on the two grayscale images (the left grayscale image with speckle structured light and the right grayscale image with speckle structured light), and then a point cloud image of the object is obtained through binocular stereo matching technology (this set of methods can be obtained using other 3D reconstruction methods, such as multi-view reconstruction, monocular vision stereo imaging technology, NERF, etc., but it is necessary to ensure that the 2D image and the 3D point cloud image are aligned, that is, C2P). The point cloud image is as follows Figure 2 shown.

[0149] 3. Use the target segmentation network V8-seg to identify the welds in the 2D image. The segmentation results are as follows: Figure 3 As shown in the figure; the red segmentation area is the segmentation area obtained by the algorithm, that is, the area where the weld is located (weld segmentation area), and the outer red box is the actual minimum closure box obtained based on the weld segmentation area.

[0150] 4. If Figure 4 As shown, the 2D image data is divided into three areas according to the circumscribed minimum closure of the identified area (the actual minimum closure box): the upper parent material area (the black area in the figure), the middle area (the gray area in the figure), and the lower parent material area (the white area in the figure). The figure is named box_mask.

[0151] 5. If Figure 5 As shown, the boundary points are obtained according to the boundary line of the external minimum closure, and the figure is named points_mask.

[0152] 6. According to the figure box_mask( Figure 4 ) and the point cloud map, the point cloud data is also divided into three areas, such as Figure 6 As shown, blue is the upper surface point cloud data, green is the weld area point cloud data, and red is the lower surface point cloud data.

[0153] 7. According to the figure points_mask ( Figure 5 ) and the corresponding relationship between the point cloud map, and obtain the boundary point cloud, such as Figure 7 shown.

[0154] 8. By integrating the RANSAC algorithm and the least squares surface fitting algorithm, the point cloud data of the upper and lower surfaces are fitted. The fitting results are as follows: Figure 8 As shown, the red point set is the base material point cloud on the weld, the green point set is the weld area point cloud, the blue point set is the base material point cloud under the weld, the dark yellow area is the extended point cloud after surface fitting of the base material point cloud on the weld, and the light blue area is the extended point cloud after surface fitting of the base material point cloud under the weld. The light blue area and the dark yellow area are the results after plane extension.

[0155] 9. Based on the surface extended from the plane, the coordinates of the boundary points of the upper surface on the lower surface can be calculated. By calculating the height difference between the coordinate points obtained by fitting calculation and the original boundary coordinate points of the lower surface, the height difference between the two planes can be obtained. The value of this height difference is the value of the misalignment. According to the size of this value, it can be determined whether there is a misalignment on the scanned surface.

[0156] 10. Figure 9 As shown, if there is no misalignment on the scanned surface, the distance to the upper and lower surfaces is calculated directly based on the purple points. The vertical distance is the residual height value of the upper and lower surfaces. The red and blue areas in the figure are both planar parent material areas (composed of the parent material point cloud above the weld and the parent material point cloud below the weld, respectively), and there is no misalignment. In this figure, the red point set is the parent material point cloud above the weld, the green point set is the weld area point cloud, and the blue point set is the parent material point cloud below the weld. The dark yellow area is the extended point cloud after surface fitting of the parent material point cloud above the weld, and the light blue area is the extended point cloud after surface fitting of the parent material point cloud below the weld. The purple point is the point with the maximum Z value found.

[0157] 11. If there is a misalignment on the scanned surface, such as Figure 10 As shown (in the figure, the red point set is the parent material point cloud on the weld, the green point set is the weld area point cloud, the blue point set is the parent material point cloud under the weld, the dark yellow area is the extended point cloud after the parent material point cloud surface fitting on the weld, the light blue area is the extended point cloud after the parent material point cloud surface fitting under the weld, and the purple point is the point with the maximum Z value found). The light blue and dark yellow areas in the figure are the extended point sets of the lower and upper surfaces. The two surfaces have obvious height differences. According to the boundary points of the upper and lower surfaces, the corresponding points of the upper and lower surfaces are connected, and the straight line points are generated between the two points based on the corresponding points based on cubic sampling interpolation. After the interpolation of all corresponding points is completed, the plane fitting algorithm that integrates RANSAC and least squares method is used to fit the plane data, calculate the vertical distance from all points in the weld area to the plane, and take the maximum value point as the point where the residual height is located, as shown in the figure. Figure 11 As shown, (in this figure, the red point set is the upper boundary point cloud of the weld, the blue point set is the lower boundary point cloud of the weld, the green point set is the weld area point cloud, the light blue point is the point with the largest z value in the weld area point cloud, and the purple point is the point with the smallest z value in the weld area point cloud).

[0158] 12. According to the position of the maximum coheight value, calculate the inverse position of the position on the initial image, and make a rectangular frame with a length of 50 pixels based on the position and draw it in the image, such as Figure 12 As shown (the red boxed area in the figure is the weld segmented by the segmentation algorithm, and the green boxed area is the area with the maximum reinforced height).

[0159] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the scope of protection of the present invention.

Claims

1. An automatic weld reinforcement detection method integrating point cloud and image segmentation, characterized by: The specific steps include: 1) Use a binocular structured light camera to shoot the object to be measured, and use binocular stereo vision technology to obtain the aligned grayscale image and point cloud image; 2) Perform target segmentation on the grayscale image, find the weld in the grayscale image, and segment the weld from the background to obtain a weld segmentation binary mask image; 3) Generate a position area mask based on the weld segmentation binary mask map, and further divide it into the upper parent material area, weld area, and lower parent material area. Generate a regional mask map and name it box_mask. It is used in conjunction with the point cloud map corresponding to the original grayscale image. 4) Based on the region mask map, extract the upper and lower boundary lines of the weld from the minimum closure region to generate the weld boundary mask map: 5) Map the regional mask map and weld boundary mask map to the point cloud map, and obtain the point cloud data and point cloud boundaries of the sub-regions; 6) By integrating RANSAC and the least squares method, a surface fitting method is used to fit the base material point cloud below the weld and obtain the lower surface equation. The X and Y values of the upper boundary point cloud of the weld are then substituted into the equation to calculate the corresponding Z value at the X and Y points. This value is then subtracted from the initial Z value of the upper boundary point cloud of the weld to obtain the height difference of the fitted point cloud. This is used to determine whether there is a misalignment during the welding process. When the mean value of the point cloud height difference is greater than 1 mm, it is determined that there is a misalignment; when the mean value of the point cloud height difference is less than 1 mm, it is determined that there is no misalignment. 7) If there is no misalignment, the weld area point cloud is denoised based on density and distance statistics. The point with the highest Z value in the denoised weld area point cloud is then selected as the highest point of the weld. The vertical distances from the highest point of the weld to the upper and lower surface equations obtained by surface fitting are calculated respectively. The maximum value of the two is finally taken as the weld height value. 8) If there is a misalignment, the points in the weld upper boundary point cloud and the weld lower boundary point cloud are matched one by one. For each pair of corresponding weld upper boundary point cloud and weld lower boundary point cloud, three sampling interpolation is performed along the slope direction of the line connecting the two points to generate all interpolation point sets; after all interpolation is completed, a surface fitting method that integrates RANSAC and least squares method is used to perform plane fitting on these interpolation point sets to obtain the plane equation; 9) After step 8), perform point cloud denoising on the weld area point cloud based on density and distance statistics. Using the plane equation obtained in step 8), calculate the vertical distance from each weld point to the plane fitted in step 8) using the point-to-plane distance formula. Take the maximum distance value as the weld reinforcement value.

2. The method for automatically detecting weld reinforcement height by integrating point cloud and image segmentation according to claim 1, characterized in that: The step 1) includes the following specific steps: 1.1) The left and right grayscale images are acquired through the left and right viewing angles of the binocular structured light camera, respectively. The left and right grayscale images together constitute a grayscale image. 1.2) Perform epipolar correction on the left and right grayscale images to ensure that they are aligned on the same horizontal line, thus obtaining the aligned grayscale images. 1.3) After step 1.2), a stereo matching algorithm is used to obtain a disparity map; 1.4) Combining the disparity map with the camera's intrinsic and extrinsic parameters, triangulate each pixel in the left and right grayscale images to its coordinate point in 3D space. 1.5) Assign the grayscale value or color information of each pixel to the corresponding 3D point, forming a one-to-one correspondence between "pixel and point cloud", thereby obtaining a point cloud image with the same resolution as the grayscale image.

3. The method for automatically detecting weld reinforcement height by integrating point cloud and image segmentation according to claim 1, characterized in that: In step 2), when performing target segmentation, the target segmentation network V8-seg is used to complete the segmentation, and the weld segmentation area and the actual minimum closure box obtained according to the weld segmentation area are obtained.

4. The method for automatically detecting weld reinforcement height by integrating point cloud and image segmentation according to claim 1, characterized in that: The step 3) includes the following steps: 3.1) Use the contour extraction method to process the weld segmentation binary mask image to obtain the preliminary minimum closure area; 3.2) Mapping the preliminary minimum closure region to an image of the same size as the initial grayscale image to generate a region mask image. On the region mask image, the weld region is marked as a middle region, an upper base material region, and a lower base material region. 3.3) After step 3.2), the generated region mask image is named "box_mask" and used in conjunction with the point cloud image corresponding to the original grayscale image.

5. The method for automatically detecting weld reinforcement height by integrating point cloud and image segmentation according to claim 4, characterized in that: The step 3.1) includes the following specific steps: 3.1.1) First, obtain the contour of the weld mask and use the minimum closure algorithm to obtain the four corner points of the contour; 3.1.2) Select the upper left and upper right corners as the starting and ending points, and interpolate according to the slope between the two points to obtain the upper boundary of the minimum closure; 3.1.3) Select the lower left and lower right corners as the starting and ending points, and interpolate according to the slope between the two points to obtain the lower boundary of the minimum closure; 3.1.4) Using the upper left corner and lower left corner as the starting and ending points, interpolate according to the slope between the two points to obtain the left boundary of the minimum closure; 3.1.5) Using the upper right corner and lower right corner as the starting and ending points, interpolate according to the slope between the two points to obtain the right boundary of the minimum closure; 3.1.6) After steps 3.1.1) to 3.1.5), the complete preliminary minimum closure region is finally obtained.

6. The method for automatically detecting weld reinforcement height by integrating point cloud and image segmentation according to claim 1, characterized in that: The step 4) includes the following specific steps: 4.1) In box_mask, use the four endpoints of the minimum closure to select the upper left corner and the upper right corner, interpolate according to the slope between the two points, obtain the upper boundary line, and generate the weld upper boundary mask; 4.2) Select the lower left corner point and the lower right corner point, interpolate according to the slope between the two points, obtain the lower boundary line and generate the weld lower boundary mask; 4.3) After steps 4.1) and 4.2), a weld boundary mask image is obtained.

7. The method for automatically detecting weld reinforcement height by integrating point cloud and image segmentation according to claim 6, characterized in that: The step 5) includes the following steps: 5.1) Map the pixel indices of the region mask image to the point cloud image with the same index (u, v) so that a corresponding "region label" is added to each point in the point cloud image; 5.2) Map the coordinate index (u, v) corresponding to each pixel in the weld boundary mask image to the point cloud image to obtain the weld boundary point cloud in three-dimensional space; 5.3) Group the points in the point cloud according to the location of the "region label": Base material point cloud on weld: all 3D points under the "region label" in the base material area; Weld area point cloud: 3D points under the "Area Label" in the middle area; Base material point cloud under the weld: 3D points under the "Area Label" in the base material area; Weld upper boundary point cloud: 3D points under the "region label" at the weld upper boundary mask; Weld lower boundary point cloud: 3D points under the "region label" at the weld lower boundary mask.

8. The method for automatically detecting weld reinforcement height by integrating point cloud and image segmentation according to claim 1, characterized in that: In step 7), the vertical distances from the highest point of the weld to the upper and lower surface equations obtained by surface fitting are calculated respectively, and the maximum value of the two is finally taken as the welding height value. The specific process is as follows: 7.1) Perform surface fitting on the base material point cloud below the weld and obtain the lower surface equation. Perform surface fitting on the base material point cloud above the weld and obtain the upper surface equation. 7.2) Substitute the X and Y values of the highest point of the weld into the upper surface equation to calculate the corresponding Z value at X and Y. Calculate the difference between the Z value of the highest point of the weld and the calculated Z value of the upper surface, and record this difference as the residual height value. 7.3) Substitute the X and Y values of the highest point of the weld into the equation of the lower surface to calculate the corresponding Z value at X and Y. Calculate the difference between the Z value of the highest point of the weld and the calculated Z value of the lower surface, and record the difference as the other residual height value; 7.4) Take the maximum of the two reinforcement values as the final reinforcement value of the weld.

Images