Automatic weld reinforcement detection method fusing point cloud and image segmentation

Abstract

The invention relates to the field of special equipment surplus height detection technology and the like, and discloses a weld surplus height automatic detection method fusing point cloud and image segmentation, which can quickly measure misalignment surplus height and non-misalignment surplus height of a weld joint by combining binocular stereoscopic vision, AI analysis and point cloud analysis technologies, make up for the defects of the traditional detection method in efficiency and accuracy, and improve the detection accuracy of the weld joint. A reliable technical means is provided for actual detection; comprising the following specific steps: shooting a measured object through a binocular structured light camera, and obtaining a grey-scale map and a point cloud map after alignment through a binocular stereoscopic vision technology; target segmentation is carried out on the grey-scale map, a welding seam in the grey-scale map is found, the welding seam is segmented from the background, and the welding seam is segmented into a binary mask map; and generating a position area mask according to the welding seam segmentation binary mask image, further dividing the position area mask into an upper base metal area, a welding seam area and a lower base metal area, generating an area mask image, naming the area mask image as boxmask, and performing combined use with a point cloud image corresponding to the original grey-scale image.

Description

Technical Field

[0001] The present invention relates to the field of special equipment reinforcement height detection technology, etc. Specifically, it is an automatic weld reinforcement height detection method that integrates point cloud and image segmentation. Background Technique

[0002] Welding technology is widely used in the field of special equipment manufacturing, such as boilers, amusement facilities, etc. The weld quality of these equipment directly affects their safety and reliability. As an important index for evaluating the weld forming quality, the weld reinforcement height can reflect the protrusion degree of the weld in the vertical direction, and thus affect the structural strength and service life of the equipment.

[0003] The existing weld reinforcement height detection methods mainly include manual visual inspection, two-dimensional image processing, contact measurement equipment, and laser scanning and point cloud analysis, etc. Manual inspection relies on the experience of operators, and has problems such as strong subjectivity and low detection efficiency; although the two-dimensional image processing method improves a certain degree of automation, due to the lack of depth information, it is difficult to accurately reflect the three-dimensional shape of the weld, and the detection accuracy is limited; contact measurement equipment such as coordinate measuring machines has high accuracy, but the equipment cost is high and the operation is complex, making it difficult to be popularized and applied in large-scale production; the laser scanning and point cloud analysis method can provide relatively comprehensive three-dimensional information, but the equipment price is expensive, the data processing is complex, and the working environment requirements are high, which limits its application in actual production.

[0004] Therefore, there is an urgent need for an automatic weld reinforcement height detection method that can integrate point cloud and image segmentation technology to improve the detection accuracy and efficiency, reduce costs, and meet the actual needs in special equipment manufacturing. Summary of the Invention

[0005] The purpose of the present invention is to provide an automatic weld reinforcement height detection method that integrates point cloud and image segmentation. By combining binocular stereo vision, AI analysis, and point cloud analysis technologies, this method can quickly measure the misalignment reinforcement height and non-misalignment reinforcement height of the weld, make up for the deficiencies of traditional detection methods in terms of efficiency and accuracy, and provide a reliable technical means for actual detection.

[0006] The present invention is realized through the following technical solutions: An automatic weld reinforcement height detection method that integrates point cloud and image segmentation, including the following specific steps: 1) Photograph the object to be measured through a binocular structured light camera, and obtain the aligned grayscale image and point cloud image through binocular stereo vision technology; 2) Perform target segmentation on the grayscale image, find the weld in the grayscale image, and segment the weld from the background to obtain a weld segmentation binary mask image; 3) Generate a position area mask according to the binary mask map segmented by the weld seam, and further divide it into the upper base material area, the weld seam area, and the lower base material area to generate an area mask map, and name the area mask map box_mask, and use it jointly with the point cloud map corresponding to the original grayscale map; 4) Based on the area mask map (box_mask), extract the upper and lower boundary lines of the weld seam from the minimum closed area to generate a weld seam boundary mask map (the upper weld seam boundary mask and the lower weld seam boundary mask): 5) Map the area mask map and the weld seam boundary mask map to the point cloud map, and obtain the point cloud data and point cloud boundary of the divided areas; 6) Through a surface fitting method that combines RANSAC and the least squares method, perform surface fitting on the lower base material point cloud of the weld seam to obtain the lower surface equation, and then substitute the X and Y values of the upper weld seam boundary point cloud into this equation to calculate the corresponding Z value at X and Y, and take the difference from the initial Z value of the upper weld seam boundary point cloud to obtain the fitting point cloud height value difference, which is used to judge whether there is edge misalignment during the welding process; when the average value of the point cloud height difference is greater than 1 mm, it is judged that there is edge misalignment; when the average value of the point cloud height difference is less than 1 mm, it is judged that there is no edge misalignment; 7) If there is no edge misalignment, perform denoising processing on the point cloud of the weld seam area based on density and distance statistics. Subsequently, select the point with the highest Z value in the denoised point cloud of the weld seam area as the highest point of the weld seam, and calculate the distances from the highest point of the weld seam to the upper and lower surface equations obtained by surface fitting in the vertical direction (Z direction) respectively. Finally, take the maximum value of the two as the reinforcement height value of the welding; 8) If there is edge misalignment, make the points in the upper weld seam boundary point cloud and the lower weld seam boundary point cloud correspond one by one. For each pair of corresponding upper weld seam boundary point cloud and lower weld seam boundary point cloud, perform cubic sampling interpolation along the slope direction of the line connecting the two points to generate all interpolation point sets; after all interpolations are completed, use a surface fitting method that combines RANSAC and the least squares method to perform plane fitting on these interpolation point sets to obtain a plane equation; 9) After step 8), perform point cloud denoising on the point cloud of the weld seam area based on density and distance statistics, and use the plane equation obtained in this step 8) to calculate the vertical distance from each weld seam point to the plane fitted in step 8) through the point-to-plane distance formula, and take the maximum distance value as the reinforcement height value of the welding.

[0007] To better implement the automatic detection method for the reinforcement height of the weld seam that combines point cloud and image segmentation according to the present invention, the following setting method is particularly adopted: The step 1) includes the following specific steps: 1.1) Obtain the left grayscale map and the right grayscale map through the left view and the right view of the binocular structured light camera respectively. The left grayscale map and the right grayscale map together constitute the grayscale map; 1.2) Rectify the left grayscale image and the right grayscale image to ensure that they are aligned on the same horizontal line, and obtain the aligned grayscale images; 1.3) After step 1.2), use a stereo matching algorithm (such as Block Matching, Semi-Global Matching, etc.) to obtain a disparity map; 1.4) Combine the disparity map with the internal and external parameters of the camera (including parameters such as the camera baseline, focal length, and optical center), and through triangulation, convert the corresponding pixels on the left grayscale image and the right grayscale image into their coordinate points in three-dimensional space; 1.5) Assign the grayscale value or color information of each pixel to the corresponding three-dimensional point to form a one-to-one correspondence between "pixel - point cloud", thereby obtaining a point cloud map with the same resolution as the grayscale image.

[0008] To better implement the automatic detection method for weld reinforcement that fuses point cloud and image segmentation described in the present invention, the following setting method is particularly adopted: In step 2), when performing target segmentation, it is completed using the target segmentation network V8-seg, and the weld segmentation area and the actual minimum bounding box obtained based on the weld segmentation area are obtained.

[0009] To better implement the automatic detection method for weld reinforcement that fuses point cloud and image segmentation described in the present invention, the following setting method is particularly adopted: Step 3) includes the following steps: 3.1) Use a contour extraction method (such as the minimum bounding rectangle / minimum closure algorithm in OpenCV) to process the weld segmentation binary mask image to obtain a preliminary minimum closure area; 3.2) Map the preliminary minimum closure area to an image with the same size as the original grayscale image to generate a region mask image. On this region mask image (which can be regarded as a marking image with the same size as the original grayscale image), mark the weld area as the middle area, the upper base metal area, and the lower base metal area; 3.3) After step 3.2), name the generated region mask image "box_mask" and use it jointly with the point cloud map corresponding to the original grayscale image.

[0010] To better implement the automatic detection method for weld reinforcement that fuses point cloud and image segmentation described in the present invention, the following setting method is particularly adopted: Step 3.1) includes the following specific steps: 3.1.1) First, obtain the contour of the weld mask and use the minimum closure algorithm to obtain the four corner points of the contour; 3.1.2) Select the upper left corner and the upper right corner as the starting point and the ending point, and interpolate according to the slope between the two points to obtain the upper boundary of the minimum closure; 3.1.3) Select the lower left corner and the lower right corner as the starting point and the ending point, and interpolate according to the slope between the two points to obtain the lower boundary of the minimum closure; 3.1.4) Take the upper left corner and the lower left corner as the starting point and the ending point, and interpolate according to the slope between the two points to get the left boundary of the minimum closure; 3.1.5) Take the upper right corner and the lower right corner as the starting and ending points, and interpolate according to the slope between the two points to obtain the right boundary of the minimum closure; 3.1.6) After steps 3.1.1) to 3.1.5), the complete preliminary minimum closure area is finally obtained.

[0011] In order to further better realize the automatic detection method of weld residual height by integrating point cloud and image segmentation described in the present invention, the following setting is particularly adopted: the step 4) includes the following specific steps: 4.1) In box_mask, use the four endpoints of the minimum closure to select the upper left corner and the upper right corner, interpolate according to the slope between the two points, obtain the upper boundary line and generate the weld upper boundary mask; 4.2) Select the lower left corner point and the lower right corner point, interpolate according to the slope between the two points, obtain the lower boundary line and generate the weld lower boundary mask; 4.3) After steps 4.1) and 4.2), a weld boundary mask image is obtained.

[0012] In order to further better realize the automatic detection method of weld residual height by integrating point cloud and image segmentation described in the present invention, the following setting is particularly adopted: the step 5) includes the following steps: 5.1) Since each pixel in the grayscale image corresponds one-to-one to the point cloud, the pixel index of the region mask image (box_mask) is mapped to the point cloud image with the same index (u, v) so as to add a corresponding "region label" to each point in the point cloud image; for example: Label 0: parent material point cloud in the weld area; Label 1: point cloud of weld area; Label 2: Point cloud of base material in the area under the weld.

[0013] 5.2) Since the boundary mask contains the pixel information of the upper and lower boundary lines of the weld, the coordinate index (u, v) corresponding to each pixel in the weld boundary mask image is mapped to the point cloud image to obtain the weld boundary point cloud in three-dimensional space; 5.3) Group the points in the point cloud according to the location of the "region label": Parent material point cloud on the weld: All three-dimensional points located under the "region label" in the upper parent material area; Weld region point cloud: Three-dimensional points under the "region label" in the middle region; Parent material point cloud under the weld: Three-dimensional points under the "region label" in the lower parent material area; Upper boundary point cloud of the weld: Three-dimensional points under the "region label" in the upper boundary mask of the weld; Lower boundary point cloud of the weld: Three-dimensional points under the "region label" in the lower boundary mask of the weld.

[0014] To further better implement an automatic weld reinforcement detection method that fuses point cloud and image segmentation described in the present invention, the following setting method is particularly adopted: In step 7), the distances from the highest point of the weld to the upper and lower surface equations obtained by surface fitting in the vertical direction are respectively calculated, and the specific process of finally taking the maximum value of the two as the weld reinforcement value is as follows: 7.1) Perform surface fitting on the parent material point cloud under the weld to obtain the lower surface equation, and perform surface fitting on the parent material point cloud on the weld to obtain the upper surface equation; 7.2) Substitute the X and Y values of the highest point of the weld into the upper surface equation to calculate the corresponding Z value at X and Y, and calculate the difference between the Z value of the highest point of the weld and the calculated Z value of the upper surface, and record this difference as a weld reinforcement value; 7.3) Substitute the X and Y values of the highest point of the weld into the lower surface equation to calculate the corresponding Z value at X and Y, and calculate the difference between the Z value of the highest point of the weld and the calculated Z value of the lower surface, and record this difference as another weld reinforcement value; 7.4) Take the maximum value of the two weld reinforcement values as the final weld reinforcement value.

[0015] To further better implement an automatic weld reinforcement detection method that fuses point cloud and image segmentation described in the present invention, the following setting method is particularly adopted: The grayscale image includes a left grayscale image obtained from the left view of a binocular structured light camera and a right grayscale image obtained from the right view of the binocular structured light camera; The point cloud image can also be obtained by using binocular vision stereoscopic imaging technology, multi-view reconstruction, monocular vision stereoscopic imaging technology, or nerf technology on the grayscale image, and during the process of reconstructing the grayscale image to the point cloud image, an alignment operation of 2D image to 3D point cloud image is also performed.

[0016] The specific steps of obtaining the point cloud image using binocular vision stereoscopic imaging technology are as follows: First, project projection speckle structured light on the left grayscale image and the right grayscale image to obtain the left grayscale image and the right grayscale image with speckle structured light; Then, on the two grayscale images, feature points are matched, and then the point cloud map of the object is obtained through binocular stereo matching technology.

[0017] Compared with the prior art, the present invention has the following advantages and beneficial effects: In the inspection industry of special equipment such as pressure vessels, boilers, and pipelines, the present invention can be applied to the measurement and quality assessment of weld reinforcement, and is suitable for quality control during the manufacturing process and rapid detection during regular inspections to ensure the safety and reliability of equipment operation. In boiler inspections, this method can be used for the measurement of weld reinforcement and defect assessment on the surface of boiler welds and pressure-bearing components, helping to improve the safety and stability of equipment operation.

[0018] The present invention proposes an automatic analysis method for weld reinforcement based on 3D scanned point clouds, and uses the original data for analysis, resulting in more accurate results.

[0019] During the calculation process of the present invention, through the point cloud of the initial surface, it can be judged whether there is edge misalignment through plane fitting, and the automatic calculation of the weld reinforcement with edge misalignment and without edge misalignment can be carried out.

[0020] The present invention combines 2D and 3D technologies to accelerate the analysis speed, which can be carried out on the edge deployment board, making the analysis lightweight.

[0021] The present invention uses binocular structured light stereoscopic imaging technology to form point clouds, and the accuracy can reach 0.01 mm. It directly analyzes through the scanned surface, greatly improving the analysis accuracy.

[0022] The present invention can calculate all regions of the weld, not just give the values of one region or one line.

[0023] The present invention can distinguish edge misalignment and non-edge misalignment through the fitting of the initial plane, and can construct different underlying algorithms for different situations.

[0024] The present invention is carried out in the form of 2D + 3D combination, and the algorithm can be deployed on lightweight edge devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is the actual image (grayscale image) of the object to be measured taken by the binocular structured light camera.

[0026] Figure 2 It is the point cloud map obtained after processing the grayscale image.

[0027] Figure 3 It is the result map (superposition map of the binary mask map and the original image) obtained after segmenting the grayscale image. Figure 4 It is the region mask map.

[0028] Figure 5 It is the weld boundary mask diagram.

[0029] Figure 6 It is the point cloud grouping diagram obtained after mapping the area mask diagram.

[0030] Figure 7 It is the upper and lower boundary point cloud grouping diagram obtained after mapping the weld boundary mask diagram.

[0031] Figure 8 It is the fitting and extension diagram of the point cloud surfaces of the upper and lower base materials on the misaligned edge.

[0032] Figure 9 It is the diagram of non - misaligned edge.

[0033] Figure 10 It is the overview diagram of misaligned edge excess height calculation.

[0034] Figure 11 It is the calculation diagram of the plane excess height of the misaligned edge boundary point cloud.

[0035] Figure 12 It is the 2D display diagram of the area with the maximum point cloud excess height. Detailed implementation manners

[0036] The present invention will be further described in detail below in conjunction with embodiments, but the implementation manners of the present invention are not limited thereto.

[0037] To make the purposes, technical solutions and advantages of the implementation manners of the present invention clearer, the technical solutions in the implementation manners of the present invention will be clearly and completely described below in conjunction with the drawings in the implementation manners of the present invention. Obviously, the described implementation manners are part of the implementation manners of the present invention, rather than all of the implementation manners. Based on the implementation manners in the present invention, all other implementation manners obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention. Therefore, the detailed description of the implementation manners of the present invention provided in the drawings below is not intended to limit the scope of the claimed present invention, but merely represents the selected implementation manners of the present invention. Based on the implementation manners in the present invention, all other implementation manners obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0038] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "a plurality" means two or more unless otherwise specifically defined.

[0039] Embodiment 1: The present invention designs an automatic detection method for weld reinforcement that integrates point cloud and image segmentation. By combining binocular stereo vision, AI analysis, and point cloud analysis technologies, this method can quickly measure the misalignment and non-misalignment of welds, making up for the deficiencies of traditional detection methods in terms of efficiency and accuracy, and providing a reliable technical means for actual detection. The specific steps are as follows: 1) Photograph the object to be measured using a binocular structured light camera, and obtain the aligned grayscale image and point cloud image through binocular stereo vision technology; 2) Perform target segmentation on the grayscale image to find the weld in the grayscale image, and segment the weld from the background to obtain the weld segmentation binary mask image; 3) Generate a position region mask according to the weld segmentation binary mask image, and further divide it into the upper base metal region, weld region, and lower base metal region to generate a region mask image, name the region mask image box_mask, and use it jointly with the point cloud image corresponding to the original grayscale image; 4) Based on the region mask image (box_mask), extract the upper and lower boundary lines of the weld from the minimum closed region to generate a weld boundary mask image (weld upper boundary mask and weld lower boundary mask): 5) Map the region mask image and the weld boundary mask image to the point cloud image, and obtain the sub-region point cloud data and point cloud boundary; 6) Through a surface fitting method that combines RANSAC and the least squares method, perform surface fitting on the lower base metal point cloud of the weld to obtain the lower surface equation, then substitute the X and Y values of the weld upper boundary point cloud into this equation to calculate the corresponding Z value at X and Y, and take the difference from the initial Z value of the weld upper boundary point cloud to obtain the fitting point cloud height value difference, which is used to determine whether there is misalignment during the welding process; when the average value of the point cloud height difference is greater than 1 mm, it is determined that there is misalignment; when the average value of the point cloud height difference is less than 1 mm, it is determined that there is no misalignment; 7) If there is no misalignment, perform denoising processing on the weld region point cloud based on density and distance statistics. Subsequently, select the point with the highest Z value in the denoised weld region point cloud as the highest point of the weld, and calculate the distances from the highest point of the weld to the upper and lower surface equations obtained through surface fitting in the vertical direction (Z direction) respectively. Finally, take the maximum value of the two as the weld reinforcement value; 8) If there is misalignment, make the points in the weld upper boundary point cloud and the weld lower boundary point cloud correspond one by one. For each pair of corresponding weld upper boundary point cloud and weld lower boundary point cloud, perform cubic sampling interpolation along the slope direction of the line connecting the two points to generate all interpolation point sets; after all interpolations are completed, use a surface fitting method that combines RANSAC and the least squares method to perform plane fitting on these interpolation point sets to obtain the plane equation; 9) After step 8), perform point cloud denoising on the point cloud in the weld area based on density and distance statistics, and use the plane equation obtained in this step 8). Calculate the vertical distance from each weld point to the plane fitted in step 8) through the point-to-plane distance formula, and take the maximum distance value as the reinforcement height value of the welding.

[0040] Example 2: This embodiment is further optimized on the basis of the above embodiments. The same parts as the foregoing technical solutions will not be described in detail here. Further, in order to better implement an automatic detection method for weld reinforcement height that integrates point cloud and image segmentation of the present invention, the following setting methods are particularly adopted: The step 1) includes the following specific steps: 1.1) Obtain a left grayscale image and a right grayscale image respectively through the left view and the right view of the binocular structured light camera. The left grayscale image and the right grayscale image together form a grayscale image; 1.2) Perform rectification on the left grayscale image and the right grayscale image to ensure that the left grayscale image and the right grayscale image are aligned on the same horizontal line, and obtain the aligned grayscale image; 1.3) After step 1.2), use a stereo matching algorithm (such as Block Matching, Semi-Global Matching, etc.) to obtain a disparity map; 1.4) Combine the disparity map with the internal and external parameters of the camera (including parameters such as camera baseline, focal length, and optical center), and convert each pixel on the left grayscale image and the right grayscale image into its coordinate point in three-dimensional space through triangulation; 1.5) Assign the grayscale value or color information of each pixel to the corresponding three-dimensional point to form a one-to-one correspondence of "pixel - point cloud", so as to obtain a point cloud map with the same resolution as the grayscale image.

[0041] Example 3: This embodiment is further optimized on the basis of any of the above embodiments. The same parts as the foregoing technical solutions will not be described in detail here. Further, in order to better implement an automatic detection method for weld reinforcement height that integrates point cloud and image segmentation of the present invention, the following setting methods are particularly adopted: When performing target segmentation in step 2), use the target segmentation network V8-seg to complete, and obtain the weld segmentation area and the actual minimum bounding box obtained according to the weld segmentation area.

[0042] Example 4: This embodiment is further optimized on the basis of any of the above embodiments. The same parts as the foregoing technical solutions will not be described in detail here. Further, in order to better implement an automatic detection method for weld reinforcement height that integrates point cloud and image segmentation of the present invention, the following setting methods are particularly adopted: The step 3) includes the following steps: 3.1) Process the binary mask image of the weld seam segmentation using a contour extraction method (such as the minimum bounding rectangle / minimum closure algorithm in OpenCV) to obtain a preliminary minimum closure region; 3.2) Map the preliminary minimum closure region onto an image with the same size as the original grayscale image to generate a region mask image. On this region mask image (which can be regarded as a labeling image with the same size as the original grayscale image), label the weld seam region as the middle region, the upper base metal region, and the lower base metal region; 3.3) After step 3.2), name the generated region mask image "box_mask" and use it jointly with the point cloud image corresponding to the original grayscale image.

[0043] Example 5: This example is a further optimization based on any of the above examples. The same parts as the previous technical solutions will not be elaborated here. To better implement a method for automatically detecting the weld reinforcement by fusing point cloud and image segmentation, the following setting method is specifically adopted: Step 3.1) includes the following specific steps: 3.1.1) First, obtain the contour of the weld mask and use the minimum closure algorithm to obtain the four corner points of the contour; 3.1.2) Select the upper left corner point and the upper right corner point as the starting point and the ending point, and perform interpolation according to the slope between the two points to obtain the upper boundary of the minimum closure; 3.1.3) Select the lower left corner point and the lower right corner point as the starting point and the ending point, and perform interpolation according to the slope between the two points to obtain the lower boundary of the minimum closure; 3.1.4) Use the upper left corner point and the lower left corner point as the starting point and the ending point, and perform interpolation according to the slope between the two points to obtain the left boundary of the minimum closure; 3.1.5) Use the upper right corner point and the lower right corner point as the starting point and the ending point, and perform interpolation according to the slope between the two points to obtain the right boundary of the minimum closure; 3.1.6) After steps 3.1.1) to 3.1.5), finally obtain a complete preliminary minimum closure region.

[0044] Example 6: This example is a further optimization based on any of the above examples. The same parts as the previous technical solutions will not be elaborated here. To better implement a method for automatically detecting the weld reinforcement by fusing point cloud and image segmentation, the following setting method is specifically adopted: Step 4) includes the following specific steps: 4.1) In box_mask, use the four end points of the minimum closure, select the upper left corner point and the upper right corner point, and perform interpolation according to the slope between the two points to obtain the upper boundary line and generate the weld upper boundary mask; 4.2) Select the lower left point and the lower right point, interpolate according to the slope between the two points, obtain the lower boundary line and generate the weld lower boundary mask; 4.3) After steps 4.1) and 4.2), obtain the weld boundary mask image.

[0045] Example 7: This example is further optimized on the basis of any of the above examples. The same parts as the foregoing technical solutions will not be elaborated here. Further, to better implement a method for automatically detecting the weld reinforcement by fusing point cloud and image segmentation, the following setting method is particularly adopted: Step 5) includes the following steps: 5.1) Since each pixel corresponds one-to-one in the grayscale image and the point cloud, map the pixel indices of the region mask image (box_mask) to the point cloud image with the same indices (u, v) to add corresponding "region labels" to each point in the point cloud image; For example: Label 0: Parent material point cloud in the upper region of the weld; Label 1: Point cloud in the weld region; Label 2: Parent material point cloud in the lower region of the weld.

[0046] 5.2) Since the boundary mask contains the pixel information of the upper and lower boundary lines of the weld, map the coordinate indices (u, v) corresponding to each pixel in the weld boundary mask image to the point cloud image to obtain the weld boundary point cloud in three-dimensional space; 5.3) Group the points in the point cloud image according to the position of the "region label": Parent material point cloud above the weld: All three-dimensional points under the "region label" in the upper parent material region; Point cloud in the weld region: All three-dimensional points under the "region label" in the middle region; Parent material point cloud below the weld: All three-dimensional points under the "region label" in the lower parent material region; Weld upper boundary point cloud: All three-dimensional points under the "region label" in the weld upper boundary mask; Weld lower boundary point cloud: All three-dimensional points under the "region label" in the weld lower boundary mask.

[0047] Example 8: This example is further optimized on the basis of any of the above examples. The same parts as the foregoing technical solutions will not be elaborated here. Further, to better implement a method for automatically detecting the weld reinforcement by fusing point cloud and image segmentation, the following setting method is particularly adopted: In step 7), the specific process of respectively calculating the distances from the highest point of the weld to the upper and lower surface equations obtained by surface fitting in the vertical direction and finally taking the maximum value of the two as the weld reinforcement value is as follows: 7.1) Fit the point cloud of the base material under the weld seam to obtain the lower surface equation, and fit the point cloud of the base material above the weld seam to obtain the upper surface equation; 7.2) Substitute the X and Y values of the highest point of the weld seam into the upper surface equation to calculate the corresponding Z value at X and Y, and calculate the difference between the Z value of the highest point of the weld seam and the calculated Z value of the upper surface. Record this difference as a reinforcement height value; 7.3) Substitute the X and Y values of the highest point of the weld seam into the lower surface equation to calculate the corresponding Z value at X and Y, and calculate the difference between the Z value of the highest point of the weld seam and the calculated Z value of the lower surface. Record this difference as another reinforcement height value; 7.4) Take the maximum value of the two reinforcement height values as the final reinforcement height value of the welding.

[0048] Example 9: This example is further optimized on the basis of any of the above examples. The same parts as the foregoing technical solutions will not be described in detail here. Further, to better implement a method for automatically detecting the reinforcement height of a weld seam that fuses point cloud and image segmentation, the following setting method is particularly adopted: The grayscale image includes a left grayscale image obtained from the left view of a binocular structured light camera and a right grayscale image obtained from the right view of the binocular structured light camera; The point cloud image can also be obtained by using binocular vision stereo imaging technology, multi-view reconstruction, monocular vision stereo imaging technology or nerf technology on the grayscale image. And during the process of reconstructing the grayscale image into a point cloud image, an alignment operation of 2D pictures to 3D point cloud pictures is also performed.

[0049] The specific steps for obtaining the point cloud image by using binocular vision stereo imaging technology are as follows: First, project projection speckle structured light on the left grayscale image and the right grayscale image to obtain the left grayscale image and the right grayscale image with speckle structured light; Then, perform feature point matching on the two grayscale images, and then obtain the point cloud image of the object through binocular stereo matching technology.

[0050] Example 10: A method for automatically detecting the reinforcement height of a weld seam that fuses point cloud and image segmentation includes the following steps: 1. Shoot the object to be measured by a binocular structured light camera. The actual image obtained from the left view of the binocular structured light camera is as Figure 1 shown, which is a grayscale image.

[0051] 2. On the left grayscale image and right grayscale image obtained by the binocular structured light camera in the left lens and right lens (the left grayscale image and right grayscale image constitute the grayscale image), project the projection speckle structured light to obtain the grayscale image with speckle structured light. On the two grayscale images (the left grayscale image with speckle structured light and the right grayscale image with speckle structured light), perform feature point matching, and then obtain the point cloud map of the object through the binocular stereo matching technology (this set of methods can be obtained using other 3D reconstruction methods, such as multi-view reconstruction, monocular vision stereo imaging technology, nerf, etc., but it is necessary to ensure the alignment of the 2D image and the 3D point cloud map, that is, C2P). The point cloud map is as Figure 2 shown.

[0052] 3. Use the target segmentation network V8-seg to identify the weld in the 2D image. The segmentation result is as Figure 3 shown; the red segmentation area is the segmentation area obtained by the algorithm, that is, the area where the weld is located (weld segmentation area), and the circumscribed red box is the actual minimum bounding box obtained according to the weld segmentation area.

[0053] 4. As Figure 4 shown, divide the 2D image data into three regions according to the circumscribed minimum bounding box (actual minimum bounding box) of the recognized region: the upper base metal region (the black region in the figure), the middle region (the gray region in the figure), and the lower base metal region (the white region in the figure). This figure is named box_mask.

[0054] 5. As Figure 5 shown, obtain the boundary points according to the boundary line of the circumscribed minimum bounding box. This figure is named points_mask.

[0055] 6. According to the corresponding relationship between the figure box_mask ( Figure 4 ) and the point cloud map, divide the point cloud data into three regions as well. As Figure 6 shown, where the blue is the upper surface point cloud data, the green is the weld area point cloud data, and the red is the lower surface point cloud data.

[0056] 7. According to the corresponding relationship between the figure points_mask ( Figure 5 ) and the point cloud map, obtain the boundary point cloud as Figure 7 shown.

[0057] 8. Through the surface fitting algorithm that combines the RANSAC algorithm and the least squares method, fit the point cloud data of the upper and lower surfaces. The fitting result is as Figure 8As shown in the figure, the red point set is the parent material point cloud on the weld seam, the green point set is the point cloud in the weld seam area, the blue point set is the parent material point cloud under the weld seam, the dark yellow area is the extended point cloud after the surface fitting of the parent material point cloud on the weld seam, and the light blue area is the extended point cloud after the surface fitting of the parent material point cloud under the weld seam. The light blue area and the dark yellow area are the results after plane extension.

[0058] 9. According to the surface extended from the plane, the coordinate points of the boundary points on the upper surface on the lower surface can be calculated. By calculating the height difference between the coordinate points obtained from the fitting calculation and the original boundary coordinates of the lower surface, the height difference between the two planes can be obtained. The value of this height difference is the value of the misalignment amount. According to the size of this value, it can be judged whether there is misalignment on the scanned surface.

[0059] 10. As Figure 9 shown, if there is no misalignment amount on the scanned surface, directly calculate the distances from the purple points to the upper and lower surfaces respectively, and the vertical distance is the residual height value between the upper and lower surfaces. In the figure, the red area and the blue area are both plane parent material areas (composed of the parent material point cloud on the weld seam and the parent material point cloud under the weld seam respectively), and there is no misalignment. In this figure, the red point set is the parent material point cloud on the weld seam, the green point set is the point cloud in the weld seam area, the blue point set is the parent material point cloud under the weld seam, the dark yellow area is the extended point cloud after the surface fitting of the parent material point cloud on the weld seam, the light blue area is the extended point cloud after the surface fitting of the parent material point cloud under the weld seam, and the purple point is the point with the largest Z value found.

[0060] 11. If there is misalignment on the scanned surface, as Figure 10 shown (in this figure, the red point set is the parent material point cloud on the weld seam, the green point set is the point cloud in the weld seam area, the blue point set is the parent material point cloud under the weld seam, the dark yellow area is the extended point cloud after the surface fitting of the parent material point cloud on the weld seam, the light blue area is the extended point cloud after the surface fitting of the parent material point cloud under the weld seam, and the purple point is the point with the largest Z value found), the light blue and dark yellow areas in the figure are the extended point sets of the lower surface and the upper surface, and there is an obvious height difference between the two surfaces. Then, according to the boundary points of the upper surface and the lower surface, connect the corresponding points of the upper and lower surfaces, and generate line points based on cubic sampling interpolation between the two points for all corresponding points. After all the corresponding points are interpolated, use the plane fitting algorithm that combines RANSAC and the least squares method to fit the plane data, calculate the vertical distances from all points in the weld seam area to the plane, and take the maximum value point as the point where the residual height is located. As Figure 11 shown (in this figure, the red point set is the upper boundary point cloud of the weld seam, the blue point set is the lower boundary point cloud of the weld seam, the green point set is the point cloud in the weld seam area, the light blue point is the point with the largest z value in the point cloud in the weld seam area, and the purple point is the point with the smallest z value in the point cloud in the weld seam area).

[0061] 12. According to the position of the determined maximum reinforcement height value, calculate the inverse position of this position on the initial image, and make a rectangular box with a length of 50 pixels based on this position and draw it in the image, as Figure 12 shown (in the figure, the weld seam segmented by the segmentation algorithm is in the red boxed area, and the area where the maximum reinforcement height is located is in the green boxed area).

[0062] The above are only the preferred embodiments of the present invention, and do not impose any form of limitation on the present invention. Any simple modification or equivalent change made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. An automatic detection method for weld reinforcement by fusing point cloud and image segmentation, characterized in that: It includes the following specific steps: 1) Shoot the object to be measured by a binocular structured light camera, and obtain the aligned grayscale image and point cloud image through binocular stereo vision technology; 2) Perform target segmentation on the grayscale image, find the weld in the grayscale image, and segment the weld from the background to obtain a weld segmentation binary mask image; 3) Generate a position area mask according to the weld segmentation binary mask image, and further divide it into the upper base metal area, the weld area, and the lower base metal area to generate an area mask image. Name the area mask image box_mask and use it jointly with the point cloud image corresponding to the original grayscale image; 4) Based on the area mask image, extract the upper and lower boundary lines of the weld from the minimum closure area to generate a weld boundary mask image; 5) Map the area mask image and the weld boundary mask image to the point cloud image, and obtain the sub-region point cloud data and the point cloud boundary; 6) Through a surface fitting method that combines RANSAC and the least squares method, perform surface fitting on the lower base metal point cloud of the weld to obtain the lower surface equation. Then, substitute the X and Y values of the upper boundary point cloud of the weld into this equation to calculate the corresponding Z value at X and Y, and take the difference from the initial Z value of the upper boundary point cloud of the weld to obtain the fitting point cloud height value difference, which is used to judge whether there is edge misalignment during the welding process; when the average value of the point cloud height difference is greater than 1 mm, it is judged that there is edge misalignment; when the average value of the point cloud height difference is less than 1 mm, it is judged that there is no edge misalignment; 7) If there is no edge misalignment, perform denoising processing on the weld area point cloud based on density and distance statistics. Then, select the point with the highest Z value in the denoised weld area point cloud as the highest point of the weld, and calculate the vertical distances from the highest point of the weld to the upper and lower surface equations obtained by surface fitting respectively. Finally, take the maximum value of the two as the reinforcement height value of the welding; 8) If there is edge misalignment, one-to-one correspondence is performed on the points in the upper boundary point cloud and the lower boundary point cloud of the weld. For each pair of corresponding upper boundary point cloud and lower boundary point cloud of the weld, three sampling interpolations are performed along the slope direction of the line connecting the two points to generate a complete set of interpolation points; after all interpolations are completed, use a surface fitting method that combines RANSAC and the least squares method to perform plane fitting on these sets of interpolation points to obtain a plane equation; 9) After step 8), perform point cloud denoising on the weld area point cloud based on density and distance statistics, and use the plane equation obtained in this step 8). Calculate the vertical distance from each weld point to the plane fitted in step 8) through the point-to-plane distance formula, and take the maximum distance value as the reinforcement height value of the welding; 2. The automatic detection method for weld reinforcement by fusing point cloud and image segmentation according to claim 1, characterized in that: The said step 1) includes the following specific steps: 1.1) Obtain the left grayscale image and the right grayscale image through the left view and the right view of the binocular structured light camera respectively. The left grayscale image and the right grayscale image together constitute the grayscale image; 1.2) Perform epipolar correction on the left grayscale image and the right grayscale image to ensure that the left grayscale image and the right grayscale image are aligned on the same horizontal line to obtain the aligned grayscale image; 1.3) After step 1.2), obtain a disparity map using a stereo matching algorithm; 1.4) Combine the disparity map with the internal and external parameters of the camera, and through triangulation, convert each pixel on the left grayscale image and the right grayscale image into its corresponding coordinate point in three-dimensional space; 1.5) Assign the grayscale value or color information of each pixel to the corresponding three-dimensional point, forming a one-to-one correspondence between "pixel - point cloud", so as to obtain a point cloud map with the same resolution as the grayscale image.

3. The automatic detection method for weld reinforcement by fusing point cloud and image segmentation according to claim 1, characterized in that: In step 2), when performing target segmentation, use the target segmentation network V8-seg to complete, and obtain the weld segmentation area and the actual minimum bounding box obtained according to the weld segmentation area.

4. The automatic detection method for weld reinforcement by fusing point cloud and image segmentation according to claim 1, wherein: Step 3) includes the following steps: 3.1) Use the contour extraction method to process the weld segmentation binary mask image to obtain the preliminary minimum bounding area; 3.2) Map the preliminary minimum bounding area to an image with the same size as the original grayscale image to generate a region mask image. On this region mask image, mark the weld area as the middle area, the upper base metal area, and the lower base metal area; After step 3.2), name the generated region mask image "box_mask" and use it jointly with the point cloud map corresponding to the original grayscale image.

5. The automatic detection method for weld reinforcement by fusing point cloud and image segmentation according to claim 4, characterized in that: Step 3.1) includes the following specific steps: 3.1.1) First, obtain the contour of the weld mask, and use the minimum bounding algorithm to obtain the four corner points of the contour; 3.1.2) Select the upper left corner point and the upper right corner point as the starting point and the ending point, and perform interpolation according to the slope between the two points to obtain the upper boundary of the minimum bounding; 3.1.3) Select the lower left corner point and the lower right corner point as the starting point and the ending point, and perform interpolation according to the slope between the two points to obtain the lower boundary of the minimum bounding; 3.1.4) Use the upper left corner point and the lower left corner point as the starting point and the ending point, and perform interpolation according to the slope between the two points to obtain the left boundary of the minimum bounding; 3.1.5) Use the upper right corner point and the lower right corner point as the starting point and the ending point, and perform interpolation according to the slope between the two points to obtain the right boundary of the minimum bounding; After steps 3.1.1) to 3.1.5), finally obtain the complete preliminary minimum bounding area.

6. The automatic detection method for the weld reinforcement by fusing point cloud and image segmentation according to claim 1, wherein: Step 4) includes the following specific steps: 4.1) In box_mask, use the four end points of the minimum bounding, select the upper left corner point and the upper right corner point, and perform interpolation according to the slope between the two points to obtain the upper boundary line and generate the weld upper boundary mask; 4.2) Select the lower left corner point and the lower right corner point, and perform interpolation according to the slope between the two points to obtain the lower boundary line and generate the weld lower boundary mask; After steps 4.1) and 4.2), obtain the weld boundary mask image.

7. The automatic detection method for weld reinforcement by fusing point cloud and image segmentation according to claim 6, characterized in that: Step 5) includes the following steps: 5.1) Map the pixel index of the region mask image to the point cloud image with the same index (u, v) to add the corresponding "region label" to each point in the point cloud image; 5.2) Map the coordinate index (u, v) corresponding to each pixel in the weld boundary mask image to the point cloud image to obtain the weld boundary point cloud in three-dimensional space; 5.3) Group the points in the point cloud image according to the position of the "region label": Parent material point cloud on the weld: All three-dimensional points located under the "area label" in the upper parent material area; Weld area point cloud: Three-dimensional points located under the "area label" in the middle area; Parent material point cloud under the weld: Three-dimensional points located under the "area label" in the lower parent material area; Upper boundary point cloud of the weld: Three-dimensional points located under the "area label" in the upper boundary mask of the weld; Lower boundary point cloud of the weld: Three-dimensional points located under the "area label" in the lower boundary mask of the weld.

8. The automatic detection method for weld reinforcement by fusing point cloud and image segmentation according to claim 1, characterized in that: In step 7), the specific process of separately calculating the distances from the highest point of the weld to the upper and lower surface equations obtained by surface fitting in the vertical direction and finally taking the maximum value of the two as the reinforcement height value of the weld is as follows: 7.1) Perform surface fitting on the parent material point cloud under the weld to obtain the lower surface equation, and perform surface fitting on the parent material point cloud on the weld to obtain the upper surface equation; 7.2) Substitute the X and Y values of the highest point of the weld into the upper surface equation to calculate the corresponding Z value at X and Y, and calculate the difference between the Z value of the highest point of the weld and the calculated Z value of the upper surface, and record this difference as a reinforcement height value; 7.3) Substitute the X and Y values of the highest point of the weld into the lower surface equation to calculate the corresponding Z value at X and Y, and calculate the difference between the Z value of the highest point of the weld and the calculated Z value of the lower surface, and record this difference as another reinforcement height value; 7.4) Take the maximum value of the two reinforcement height values as the final reinforcement height value of the weld.

Images