An automatic detection method for weld misalignment by integrating point cloud and image segmentation

Abstract

The present invention relates to the technical field of weld misalignment detection in special equipment, aerospace, bridges, ships, wind power, etc., such as containers, pipelines, boilers, amusement facilities, lifting machinery, passenger ropeways, etc. Disclosed is a method for automatically detecting weld misalignment by integrating point cloud and image segmentation. By combining binocular stereo vision, AI and point cloud analysis technology, the misalignment of welds can be measured quickly and accurately, making up for the shortcomings of traditional detection methods in efficiency and accuracy. The specific steps are as follows: shooting with a binocular structured light camera to obtain aligned grayscale images and point cloud images; segmenting the weld on the grayscale image, generating a position area mask based on the segmented weld, and dividing the actual object into upper weld, weld and lower weld areas; then extracting the weld boundary mask based on the segmented weld, mapping the position mask and boundary mask to the point cloud image, and obtaining regional point cloud data and boundaries.

Description

Technical Field

[0001] The present invention relates to the field of weld misalignment detection technology for special equipment, aerospace, bridges, ships, wind power, and other applications, such as containers, pipelines, boilers, amusement facilities, lifting machinery, and passenger ropeways. Specifically, it provides an automatic weld misalignment detection method that integrates point cloud and image segmentation. Background Art

[0002] Welding technology is widely used in special equipment, aerospace, bridges, ships, wind power, and other fields, such as containers, pipelines, boilers, amusement facilities, lifting machinery, and passenger ropeways. The quality of welds in these equipment directly affects their safety and reliability. Misalignment, a key indicator for evaluating weld quality, reflects the amount of misalignment between two adjacent welded parts at the joint, which in turn affects the structural strength and service life of the equipment.

[0003] When measuring the misalignment, existing methods (such as Figure 1 Typically, the method involves first placing the main ruler against one side of the weld, then sliding the height gauge against the other side of the weld. The height gauge's indication indicates the misalignment. Existing methods have the following issues: Weld gauge measurements rely heavily on operator experience, requiring visual identification of the maximum misalignment. This is time-consuming and labor-intensive, resulting in low detection efficiency. Different inspectors may produce different results, creating a high degree of subjectivity. Furthermore, a weld gauge can only measure the misalignment value at a single point at a time.

[0004] Therefore, there is an urgent need for an automatic detection method for weld misalignment that can integrate point cloud and image segmentation technology to improve the accuracy and efficiency of detection, reduce costs, and adapt to the actual needs of special equipment manufacturing. Summary of the Invention

[0005] The purpose of the present invention is to provide an automatic detection method for weld misalignment by integrating point cloud and image segmentation. This method can quickly and accurately measure the misalignment of welds by combining binocular stereo vision, AI analysis and point cloud analysis technology, making up for the shortcomings of traditional detection methods in efficiency and accuracy, and providing a reliable technical means for actual detection.

[0006] The present invention is implemented by the following technical solution: a method for automatically detecting weld misalignment by integrating point cloud and image segmentation, comprising the following specific steps:

[0007] 1) Use a binocular structured light camera to shoot the object to be measured, and use binocular stereo vision technology to obtain the aligned grayscale image and point cloud image;

[0008] 2) Perform target segmentation on the grayscale image, find the weld in the grayscale image, and separate the weld from the background;

[0009] 3) Generate a position area mask based on the segmented weld, and divide the actual object area into the weld upper area, weld area, and weld lower area;

[0010] 4) Obtain the weld boundary mask based on the segmented weld position:

[0011] 5) Map the location area mask and boundary mask to the point cloud image, and obtain the point cloud data and point cloud boundaries of the sub-areas; the point cloud data includes the point cloud of the upper area of the weld, the weld point cloud, and the point cloud of the area below the weld, and the point cloud boundaries include the upper boundary point cloud and the lower boundary point cloud;

[0012] 6) Perform bounding box cropping on all acquired point cloud data, filter the point cloud data by setting a point cloud boundary threshold, and then perform noise filtering based on the Mahalanobis distance to remove noise points in the point cloud;

[0013] 7) Perform principal component analysis (PCA) on the filtered weld point cloud to determine the principal axis direction of the weld point cloud; calculate the offset matrix between the principal axis and the X-axis, and apply this offset matrix to the transformation of the weld upper area point cloud, weld lower area point cloud, upper boundary point cloud, and lower boundary point cloud;

[0014] 8) Using a surface fitting method that integrates RANSAC and least squares, the surface of the area below the weld is fitted to the point cloud and the surface equation of the area below the weld is obtained. The surface is completed and extended along the direction from the lower boundary point cloud to the weld area. The X and Y coordinates of all points in the upper boundary point cloud are used to calculate the height values Z_fit of all points on the extended surface.

[0015] 9) According to the calculated height value Z_fit and the height value Z of the initial upper boundary point cloud, calculate the height difference Z_diff between the corresponding points, and take the maximum value of Z_diff as the maximum misalignment result between the point clouds in the upper and lower areas of the weld.

[0016] To further better implement the method for automatically detecting weld misalignment by integrating point cloud and image segmentation according to the present invention, the following setting is particularly adopted: Step 1) includes the following specific steps:

[0017] 1.1) The object under test is photographed using the left and right perspectives of a binocular structured light camera to obtain a left grayscale image and a right grayscale image, respectively. The left and right grayscale images together constitute a grayscale image.

[0018] 1.2) Projecting speckle structured light onto the left grayscale image and the right grayscale image to generate the left grayscale image and the right grayscale image with speckle structured light;

[0019] 1.3) After step 1.2), feature point matching is performed on the left and right grayscale images, and a point cloud image of the object is generated using binocular stereo matching technology.

[0020] In order to further better implement the method for automatically detecting weld misalignment by integrating point cloud and image segmentation described in the present invention, the following setting is particularly adopted: in step 2), when performing weld segmentation, the target segmentation network FastSAM after fine-tuning is used to complete the segmentation, and the weld segmentation area and the minimum closure box obtained based on the weld segmentation area are obtained.

[0021] To further better implement the method for automatically detecting weld misalignment by integrating point cloud and image segmentation according to the present invention, the following setting is particularly adopted: Step 3) includes the following steps:

[0022] 3.1) Extend the upper and lower boundaries of the minimum enclosing box to the left and right ends to the grayscale image boundary as the new upper and lower boundary lines;

[0023] 3.2) The area between the new upper and lower boundary lines is defined as the weld area. The area above the new upper boundary line is the upper weld area, and the area below the new lower boundary line is the lower weld area.

[0024] 3.3) Create a blank image, map the weld area, the area above the weld, and the area below the weld into it, and name the image box_mask.

[0025] To further better implement the method for automatically detecting weld misalignment by integrating point cloud and image segmentation according to the present invention, the following setting is particularly adopted: Step 4) includes the following specific steps:

[0026] 4.1) Extend the upper and lower boundaries of the minimum enclosing box to the left and right ends to the grayscale image boundary as the new upper and lower boundaries;

[0027] 4.2) Create a blank map, map the new upper and lower boundaries to the map, and name the map boarder_mask.

[0028] To further better implement the method for automatically detecting weld misalignment by integrating point cloud and image segmentation according to the present invention, the following setting is particularly adopted: Step 5) includes the following specific steps:

[0029] 5.1) According to the box_mask and the corresponding point cloud image, the upper weld area, weld area, and lower weld area in the box_mask are mapped to the point cloud image to obtain the upper weld area point cloud, weld area point cloud, and lower weld area point cloud;

[0030] 5.2) Based on the boarder_mask and the corresponding point cloud, map the upper and lower boundaries in the boarder_mask to the point cloud to obtain the upper and lower boundary point clouds.

[0031] To further better implement the method for automatically detecting weld misalignment by integrating point cloud and image segmentation according to the present invention, the following setting is particularly adopted: Step 6) includes the following steps:

[0032] 6.1) Perform a bounding box cropping operation on the acquired point cloud data (the point cloud above the weld, the point cloud below the weld, and the weld point cloud). Specifically, the bounding box cropping operation involves filtering the point cloud data (the point cloud above the weld, the point cloud below the weld, and the weld point cloud) using a preset point cloud boundary threshold to remove invalid or abnormal points that exceed the set spatial range, thereby preliminarily optimizing the spatial distribution of the point cloud dataset.

[0033] 6.2) After step 6.1), the initially filtered point clouds above the weld, below the weld, and weld points are each subjected to noise filtering based on the Mahalanobis distance. By calculating the statistical distance of each point relative to the point cloud data distribution, noise points that deviate from the main data distribution are identified and removed. Specifically, the Mahalanobis distance comprehensively considers the covariance structure of the point cloud data and can effectively distinguish normal points from anomalous noise points, thereby further improving the quality and accuracy of the point cloud data.

[0034] To further better implement the method for automatically detecting weld misalignment by integrating point cloud and image segmentation according to the present invention, the following setting is particularly adopted: Step 7) includes the following specific steps:

[0035] 7.1) Perform principal component analysis (PCA) on the filtered weld point cloud to calculate and determine the principal axis direction of the weld point cloud;

[0036] 7.2) After step 7.1), calculate the offset matrix between the main axis and the X axis, and apply the offset matrix to the point cloud of the upper area of the weld, the point cloud of the lower area of the weld, the upper boundary point cloud, and the lower boundary point cloud to complete the rotational alignment of the point cloud.

[0037] In order to further better realize the automatic detection method of weld misalignment by integrating point cloud and image segmentation described in the present invention, the following setting method is particularly adopted: the grayscale image includes a left grayscale image obtained by the left perspective of the binocular structured light camera and a right grayscale image obtained by the right perspective of the binocular structured light camera; the point cloud image can also be obtained by using binocular vision stereo imaging technology, multi-perspective reconstruction, monocular vision stereo imaging technology or NeRF technology to obtain the grayscale image, and in the process of reconstructing the grayscale image to the point cloud image, an alignment operation from the 2D image to the 3D point cloud image (i.e., C2P alignment) is also performed.

[0038] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0039] In the pressure vessel inspection industry, the present invention can be applied to the measurement and quality assessment of weld misalignment, and is suitable for quality control during the manufacturing process and rapid detection during regular inspections to ensure the safety and reliability of equipment operation.

[0040] In boiler inspection, the present invention can be used to measure the misalignment and defect assessment of boiler welds and pressure component surfaces, helping to improve the safety and stability of equipment operation.

[0041] In the inspection of lifting machinery, the present invention can be applied to the misalignment measurement and defect detection of compressed welds with cylindrical structures, such as the cylindrical columns of gantry cranes, the flexible legs of shipbuilding gantry cranes, the cylindrical legs and cylindrical pull rods of container gantry cranes, etc. By detecting the misalignment of the cylindrical surface, it is ensured that the above cylindrical structures do not deform or crack during use, effectively ensuring the structural stability during service.

[0042] In the inspection of amusement facilities, the present invention can be applied to the misalignment measurement and defect detection of compressed welds with cylindrical structures, such as the cylindrical columns in Ferris wheels, the cylindrical pull rods of spacecraft, and compressed tracks with cylindrical structures. By detecting the misalignment of the cylindrical surface, it can help to achieve the control of manufacturing and assembly accuracy and the dynamic assessment of fatigue damage of key welds during operation, ensuring the safety and reliability of the equipment throughout its life cycle.

[0043] In the inspection of metal and non-metallic pipelines, the present invention can be applied to the detection of misalignment of welds in pipelines of different materials. It can help identify the risk of sealing failure of high-temperature and high-pressure metal pipelines and structural deformation caused by installation misalignment of non-metallic flexible pipelines, thereby ensuring the integrity and operational reliability of the pipeline system.

[0044] In the aerospace field, the present invention can be applied to the detection of misalignment of welds in key components such as spacecraft pressure vessels and fuel storage tanks. It can help identify weld defects caused by manufacturing or assembly, ensure the structural integrity and sealing performance of welds under high temperature, high pressure and extreme environments, and ensure the safety of spacecraft launch and operation and mission reliability.

[0045] By combining binocular stereo vision, AI analysis and point cloud analysis technology, the present invention can quickly and accurately measure the misalignment of welds, making up for the shortcomings of traditional detection methods in efficiency and accuracy, and providing a reliable technical means for actual detection.

[0046] The present invention proposes an automatic analysis method for misalignment based on 3D scanning point cloud, which promotes the misalignment detection towards digital detection.

[0047] The present invention combines 2D and 3D technologies to speed up analysis and can be performed on edge deployment boards, making the analysis lightweight.

[0048] The present invention proposes an automatic analysis method for misalignment based on 3D point cloud, which reduces dependence on users, improves the repeatability and accuracy of detection, and can automatically measure the misalignment at various locations on the weld in one scan.

[0049] The binocular structured light stereo imaging technology can generate point clouds with an accuracy of 0.01 mm, and analysis can be performed directly on the scanned surface, which greatly improves the accuracy of the analysis.

[0050] The present invention can perform calculations on all areas of a weld, rather than simply providing values on one area or one line.

[0051] The present invention adopts the combination of 2D+3D, and the algorithm can be deployed on lightweight edge devices.

[0052] The algorithm of the present invention integrates 2D image recognition and 3D point cloud analysis, resulting in high recognition accuracy and repeatability. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Schematic diagram of the prior art misalignment measurement.

[0054] Figure 2 It is the actual image of the object to be measured obtained from the left perspective of the binocular structured light camera.

[0055] Figure 3 The binocular stereo matching technology is used to match feature points on the grayscale image to generate a point cloud image of the object.

[0056] Figure 4 The result of weld segmentation on the grayscale image is displayed.

[0057] Figure 5 The location area mask generated for the segmented weld.

[0058] Figure 6 Boundary mask map of the weld.

[0059] Figure 7 It is the map formed by mapping the location area mask to the point cloud map.

[0060] Figure 8 It is the map formed by mapping the boundary mask to the point cloud.

[0061] Figure 9 The result image after point cloud rotation alignment.

[0062] Figure 10 Schematic diagram of all height values Z_fit on the extended surface. DETAILED DESCRIPTION

[0063] The present invention will be further described in detail below with reference to the examples, but the embodiments of the present invention are not limited thereto.

[0064] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention for which protection is sought, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0065] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature identified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0066] Example 1:

[0067] The present invention designs an automatic detection method for weld misalignment that integrates point cloud and image segmentation. By combining binocular stereo vision, AI analysis, and point cloud analysis technology, this method can quickly and accurately measure weld misalignment, making up for the shortcomings of traditional detection methods in efficiency and accuracy, and providing a reliable technical means for actual detection. The method includes the following specific steps:

[0068] 1) Use a binocular structured light camera to shoot the object to be measured, and use binocular stereo vision technology to obtain the aligned grayscale image and point cloud image;

[0069] 2) Perform target segmentation on the grayscale image, find the weld in the grayscale image, and separate the weld from the background;

[0070] 3) Generate a position area mask based on the segmented weld, and divide the actual object area into the weld upper area, weld area, and weld lower area;

[0071] 4) Obtain the weld boundary mask based on the segmented weld position:

[0072] 5) Map the location area mask and boundary mask to the point cloud image, and obtain the point cloud data and point cloud boundaries of the sub-areas; the point cloud data includes the point cloud of the upper area of the weld, the weld point cloud, and the point cloud of the area below the weld, and the point cloud boundaries include the upper boundary point cloud and the lower boundary point cloud;

[0073] 6) Perform bounding box cropping on all acquired point cloud data, filter the point cloud data by setting a point cloud boundary threshold, and then perform noise filtering based on the Mahalanobis distance to remove noise points in the point cloud;

[0074] 7) Perform principal component analysis on the filtered weld point cloud to determine the principal axis direction of the weld point cloud; calculate the offset matrix between the principal axis and the X-axis, and apply this offset matrix to the transformation of the weld upper area point cloud, weld lower area point cloud, upper boundary point cloud, and lower boundary point cloud;

[0075] 8) A surface fitting method that integrates RANSAC and the least squares method (MLS) is used to perform surface fitting on the point cloud of the area under the weld and obtain the surface equation of the area under the weld. The surface is completed and extended along the direction from the lower boundary point cloud to the weld area. The X and Y coordinates of all points in the upper boundary point cloud are used to calculate the height values Z_fit of all points' X and Y coordinates on the extended surface.

[0076] 9) According to the calculated height value Z_fit and the height value Z of the initial upper boundary point cloud, calculate the height difference Z_diff between the corresponding points, and take the maximum value of Z_diff as the maximum misalignment result between the point clouds in the upper and lower areas of the weld.

[0077] Example 2:

[0078] This embodiment is a further optimization based on the above embodiment. The similarities with the above technical solutions are not repeated here. In order to better implement the automatic detection method of weld misalignment by integrating point cloud and image segmentation according to the present invention, the following setting is particularly adopted: Step 1) includes the following specific steps:

[0079] 1.1) The object under test is photographed using the left and right perspectives of a binocular structured light camera to obtain a left grayscale image and a right grayscale image, respectively. The left and right grayscale images together constitute a grayscale image.

[0080] 1.2) Projecting speckle structured light onto the left grayscale image and the right grayscale image to generate the left grayscale image and the right grayscale image with speckle structured light;

[0081] 1.3) After step 1.2), feature point matching is performed on the left and right grayscale images, and a point cloud image of the object is generated using binocular stereo matching technology.

[0082] Example 3:

[0083] This embodiment is further optimized based on any of the above embodiments, and the similarities with the above technical solutions are not repeated here. In order to better realize the automatic detection method of weld misalignment by integrating point cloud and image segmentation as described in the present invention, the following setting is particularly adopted: In the step 2), when performing weld segmentation, the target segmentation network FastSAM after fine-tuning is used to complete the segmentation, and the weld segmentation area and the minimum closed box obtained based on the weld segmentation area are obtained.

[0084] Example 4:

[0085] This embodiment is a further optimization based on any of the above embodiments. The similarities with the above technical solutions are not repeated here. In order to better implement the automatic detection method of weld misalignment by integrating point cloud and image segmentation according to the present invention, the following setting is particularly adopted: Step 3) includes the following steps:

[0086] 3.1) Extend the upper and lower boundaries of the minimum enclosing box to the left and right ends to the grayscale image boundary as the new upper and lower boundary lines;

[0087] 3.2) The area between the new upper and lower boundary lines is defined as the weld area. The area above the new upper boundary line is the upper weld area, and the area below the new lower boundary line is the lower weld area.

[0088] 3.3) Create a blank image, map the weld area, the area above the weld, and the area below the weld into it, and name the image box_mask.

[0089] Example 5:

[0090] This embodiment is a further optimization based on any of the above embodiments. The similarities with the above technical solutions are not repeated here. In order to better implement the automatic detection method of weld misalignment by integrating point cloud and image segmentation according to the present invention, the following setting is particularly adopted: Step 4) includes the following specific steps:

[0091] 4.1) Extend the upper and lower boundaries of the minimum enclosing box to the left and right ends to the grayscale image boundary as the new upper and lower boundaries;

[0092] 4.2) Create a blank map, map the new upper and lower boundaries to the map, and name the map boarder_mask.

[0093] Example 6:

[0094] This embodiment is a further optimization based on any of the above embodiments. The similarities with the above technical solutions are not repeated here. In order to better implement the automatic detection method of weld misalignment by integrating point cloud and image segmentation according to the present invention, the following setting is particularly adopted: Step 5) includes the following specific steps:

[0095] 5.1) According to the box_mask and the corresponding point cloud image, the upper weld area, weld area, and lower weld area in the box_mask are mapped to the point cloud image to obtain the upper weld area point cloud, weld area point cloud, and lower weld area point cloud;

[0096] 5.2) Based on the boarder_mask and the corresponding point cloud, map the upper and lower boundaries in the boarder_mask to the point cloud to obtain the upper and lower boundary point clouds.

[0097] Example 7:

[0098] This embodiment is a further optimization based on any of the above embodiments. The similarities with the above technical solutions are not repeated here. In order to better implement the automatic detection method of weld misalignment by integrating point cloud and image segmentation according to the present invention, the following setting is particularly adopted: Step 6) includes the following steps:

[0099] 6.1) Perform a bounding box cropping operation on the acquired point cloud data (the point cloud above the weld, the point cloud below the weld, and the weld point cloud). Specifically, the bounding box cropping operation involves filtering the point cloud data (the point cloud above the weld, the point cloud below the weld, and the weld point cloud) using a preset point cloud boundary threshold to remove invalid or abnormal points that exceed the set spatial range, thereby preliminarily optimizing the spatial distribution of the point cloud dataset.

[0100] 6.2) After step 6.1), the initially filtered point clouds above the weld, below the weld, and weld points are each subjected to noise filtering based on the Mahalanobis distance. By calculating the statistical distance of each point relative to the point cloud data distribution, noise points that deviate from the main data distribution are identified and removed. Specifically, the Mahalanobis distance comprehensively considers the covariance structure of the point cloud data and can effectively distinguish normal points from anomalous noise points, thereby further improving the quality and accuracy of the point cloud data.

[0101] Example 8:

[0102] This embodiment is a further optimization based on any of the above embodiments. The similarities with the above technical solutions are not repeated here. In order to better implement the automatic detection method of weld misalignment by integrating point cloud and image segmentation according to the present invention, the following setting is particularly adopted: Step 7) includes the following specific steps:

[0103] 7.1) Perform principal component analysis (PCA) on the filtered weld point cloud to calculate and determine the principal axis direction of the weld point cloud;

[0104] 7.2) After step 7.1), calculate the offset matrix between the main axis and the X axis, and apply the offset matrix to the point cloud of the upper area of the weld, the point cloud of the lower area of the weld, the upper boundary point cloud, and the lower boundary point cloud to complete the rotational alignment of the point cloud.

[0105] Example 9:

[0106] This embodiment is further optimized on the basis of any of the above embodiments, and the similarities with the above technical solutions are not repeated here. In order to further better realize the automatic detection method of weld misalignment by integrating point cloud and image segmentation as described in the present invention, the following setting method is particularly adopted: the grayscale image includes a left grayscale image obtained by the left perspective of the binocular structured light camera and a right grayscale image obtained by the right perspective of the binocular structured light camera; the point cloud image can also be obtained by using binocular vision stereo imaging technology, multi-perspective reconstruction, monocular vision stereo imaging technology or NeRF technology to convert the grayscale image into a point cloud image, and in the process of reconstructing the grayscale image into the point cloud image, an alignment operation from a 2D image to a 3D point cloud image (i.e., C2P alignment) is also performed.

[0107] Example 10:

[0108] A method for automatically detecting weld misalignment by integrating point cloud and image segmentation comprises the following steps:

[0109] The first step is to shoot the object under test through the left and right perspectives of the binocular structured light camera, and obtain the left grayscale image (such as Figure 2 As shown) and the right grayscale image; speckle structured light is projected onto the left grayscale image and the right grayscale image to generate a left grayscale image and a right grayscale image with speckle structured light; then, feature point matching is performed on the left grayscale image and the right grayscale image, and the binocular stereo matching technology is used to generate a point cloud image of the object under test. The left grayscale image obtained by the left perspective of the binocular structured light camera and the right grayscale image obtained by the right perspective of the binocular structured light camera together constitute a grayscale image. The point cloud image can also be obtained by using binocular vision stereo imaging technology, multi-perspective reconstruction, monocular vision stereo imaging technology or NeRF technology to reconstruct the grayscale image. In the process of reconstructing the grayscale image to the point cloud image, an alignment operation from 2D image to 3D point cloud image (ie C2P alignment) is also performed. The point cloud image is shown in FIG. Figure 3 shown.

[0110] In the second step, the fine-tuned target segmentation network FastSAM is used to segment the target on the grayscale image (2D image). The weld in the grayscale image is found and separated from the background. The weld segmentation area and the minimum closure box obtained based on the weld segmentation area are obtained. The segmentation result is shown in the figure below. Figure 4 As shown in the figure, the red segmentation area is the segmentation area obtained by the algorithm, that is, the area where the weld is located, and the circumscribed red box is the actual minimum closure box obtained based on the segmentation area.

[0111] The third step is to divide the 2D grayscale image into three regions according to the minimum closure frame obtained in the second step: Figure 4 The upper and lower boundaries of the red box are extended to the left and right ends to the image boundaries as the new upper and lower boundary lines; the area between the new upper and lower boundary lines is defined as the weld area, the area above the new upper boundary line is the upper weld area, and the area below the new lower boundary line is the lower weld area. Create a blank image and map the above three areas into it, such as Figure 5 As shown, the area above the weld is mapped to black, the weld area is mapped to gray, and the area below the weld is mapped to white, and the map is named box_mask.

[0112] Step 4: Based on the minimum closure box obtained in step 4) of step 3, Figure 4 The upper and lower boundaries of the red box are extended to the left and right ends to the image boundary as the new upper and lower boundaries. Create a blank image and map the new upper and lower boundaries to the image, such as Figure 6As shown, pure white is the upper boundary and grayish white is the lower boundary. The image is named boarder_mask.

[0113] Step 5. According to the box_mask and the corresponding point cloud map, the weld upper area, weld area, and weld lower area in the box_mask are mapped to the point cloud map to obtain the weld upper area point cloud, weld point cloud, and weld lower area point cloud, as shown in the figure. Figure 7 As shown, the blue point set is the point set in the upper area of the weld, the green area is the weld click, and the red area is the point set in the lower area of the weld.

[0114] Step 6: According to the boarder_mask and the corresponding point cloud, map the upper and lower boundaries in the boarder_mask to the point cloud to obtain the upper and lower boundary point clouds. Figure 8 As shown, the green point set is the upper boundary point cloud, and the red point set is the lower boundary point cloud.

[0115] Step 7: Perform a bounding box cropping operation on the weld upper area point cloud, weld lower area point cloud, and weld point cloud obtained in step 5. The bounding box cropping operation is specifically as follows: the point cloud data (weld upper area point cloud, weld lower area point cloud, weld point cloud) are filtered by a preset point cloud boundary threshold to eliminate invalid points or abnormal points that exceed the set spatial range, thereby preliminarily optimizing the spatial distribution of the point cloud dataset.

[0116] Step 8: Perform noise filtering based on the Mahalanobis distance on the initially filtered point clouds above the weld, below the weld, and weld point clouds. By calculating the statistical distance of each point relative to the point cloud data distribution, noise points that deviate from the main data distribution are identified and removed. Specifically, the Mahalanobis distance comprehensively considers the covariance structure of the point cloud data and can effectively distinguish normal points from abnormal noise points, thereby further improving the quality and accuracy of the point cloud data.

[0117] Step 9: Perform principal component analysis (PCA) on the filtered weld point cloud to determine the principal axis direction of the weld point cloud; then calculate the offset matrix between the principal axis and the X axis, and apply the offset matrix to the weld upper area point cloud, weld lower area point cloud, upper boundary point cloud, and lower boundary point cloud to complete the rotation alignment of the point cloud. The result after rotation is as follows: Figure 9 As shown in the figure, the blue area is the point set of the upper area of the weld, the red area is the point set of the lower area of the weld, and the green area is the point set of the weld area. In the figure, the red, blue, and green coordinate axes represent the X-axis, Y-axis, and Z-axis in the world coordinate system, respectively.

[0118] Step 10: Use a surface fitting method that combines RANSAC and the least squares method (MLS) to perform surface fitting on the point cloud under the weld. Based on the fitting, the surface equation of the area under the weld is obtained. The surface is completed and extended along the direction from the lower boundary point cloud to the weld area. The X and Y coordinates of all points in the upper boundary point cloud are used to calculate the height values Z_fit of all points on the extended surface. The result is shown in the figure below. Figure 10 shown. Figure 10 Includes a top view (left) and a side view (right). The red point set represents the point cloud above the weld, the gray point set represents the point cloud below the weld, and the green point set represents the result of fitting the surface of the point cloud below the weld and extending it along the weld area. In the top view, the blue point set represents the upper boundary point cloud; in the side view, the yellow point set represents the point set calculated by substituting the X and Y coordinates of the upper boundary point cloud into the surface equation of the region below the weld.

[0119] Step 11. Calculate the height difference Z_diff between the corresponding points based on the calculated height value Z_fit and the height value Z of the initial upper boundary point cloud, and take the maximum value of Z_diff as the maximum misalignment result between the point clouds in the upper and lower areas of the weld.

[0120] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the scope of protection of the present invention.

Claims

1. A method for automatically detecting weld misalignment by integrating point cloud and image segmentation, characterized by: The specific steps include: 1) Use a binocular structured light camera to shoot the object to be measured, and use binocular stereo vision technology to obtain the aligned grayscale image and point cloud image; 2) Perform target segmentation on the grayscale image, find the weld in the grayscale image, and separate the weld from the background. When performing weld segmentation, use the fine-tuned target segmentation network FastSAM to complete the task, and obtain the weld segmentation area and the minimum closed box obtained based on the weld segmentation area. 3) Generate a position area mask based on the segmented weld, and divide the actual object area into the weld upper area, weld area, and weld lower area; The steps include: 3.1) Extend the upper and lower boundaries of the minimum enclosing box to the left and right ends to the grayscale image boundary as the new upper and lower boundary lines; 3.2) The area between the new upper and lower boundary lines is defined as the weld area. The area above the new upper boundary line is the upper weld area, and the area below the new lower boundary line is the lower weld area. 3.3) Create a blank image, map the weld area, the area above the weld, and the area below the weld into it, and name the image box_mask; 4) Obtaining the weld boundary mask based on the segmented weld position; including the following specific steps: 4.1) Extend the upper and lower boundaries of the minimum enclosing box to the left and right ends to the grayscale image boundary as the new upper and lower boundaries; 4.2) Create a blank map, map the new upper and lower boundaries to the map, and name the map boarder_mask; 5) Map the location region mask and boundary mask to the point cloud image, and obtain the point cloud data and point cloud boundaries of the sub-regions; including the following specific steps: 5.1) According to the box_mask and the corresponding point cloud image, the upper weld area, weld area, and lower weld area in the box_mask are mapped to the point cloud image to obtain the upper weld area point cloud, weld area point cloud, and lower weld area point cloud; 5.2) Based on the boarder_mask and the corresponding point cloud, map the upper and lower boundaries in the boarder_mask to the point cloud to obtain the upper and lower boundary point clouds; 6) Perform bounding box cropping on all acquired point cloud data, filter the point cloud data by setting a point cloud boundary threshold, and then perform noise filtering based on the Mahalanobis distance to remove noise points in the point cloud; 7) Perform principal component analysis on the filtered weld point cloud to determine the principal axis direction of the weld point cloud; calculate the offset matrix between the principal axis and the X-axis, and apply this offset matrix to the transformation of the weld upper area point cloud, weld lower area point cloud, upper boundary point cloud, and lower boundary point cloud; 8) Using a surface fitting method that integrates RANSAC and least squares, the surface of the area below the weld is fitted to the point cloud and the surface equation of the area below the weld is obtained. The surface is completed and extended along the direction from the lower boundary point cloud to the weld area. The X and Y coordinates of all points in the upper boundary point cloud are used to calculate the height values Z_fit of all points on the extended surface. 9) According to the calculated height value Z_fit and the height value Z of the initial upper boundary point cloud, calculate the height difference Z_diff between the corresponding points, and take the maximum value of Z_diff as the maximum misalignment result between the point clouds in the upper and lower areas of the weld.

2. The method for automatically detecting weld misalignment by integrating point cloud and image segmentation according to claim 1, characterized in that: The step 1) includes the following specific steps: 1.1) The object under test is photographed using the left and right perspectives of a binocular structured light camera to obtain a left grayscale image and a right grayscale image, respectively. The left and right grayscale images together constitute a grayscale image. 1.2) Projecting speckle structured light onto the left grayscale image and the right grayscale image to generate the left grayscale image and the right grayscale image with speckle structured light; 1.3) After step 1.2), feature point matching is performed on the left and right grayscale images, and a point cloud image of the object is generated using binocular stereo matching technology.

3. The method for automatically detecting weld misalignment by integrating point cloud and image segmentation according to claim 1, characterized in that: The step 6) includes the following steps: 6.1) Performing a bounding box cropping operation on the acquired point cloud data of the upper weld area, the lower weld area, and the weld point cloud. Specifically, the bounding box cropping operation involves filtering the point cloud data using a preset point cloud boundary threshold to remove invalid or abnormal points that exceed the set spatial range, thereby preliminarily optimizing the spatial distribution of the point cloud dataset. 6.2) After step 6.1), perform noise filtering based on the Mahalanobis distance on the initially filtered point clouds of the weld upper region, weld lower region, and weld point clouds. By calculating the statistical distance of each point relative to the point cloud data distribution, noise points that deviate from the main data distribution are identified and removed.

4. The method for automatically detecting weld misalignment by integrating point cloud and image segmentation according to claim 1, characterized in that: The step 7) includes the following specific steps: 7.1) Perform principal component analysis on the filtered weld point cloud and calculate and determine the principal axis direction of the weld point cloud; 7.2) After step 7.1), calculate the offset matrix between the main axis and the X axis, and apply the offset matrix to the point cloud of the upper area of the weld, the point cloud of the lower area of the weld, the upper boundary point cloud, and the lower boundary point cloud to complete the rotational alignment of the point cloud.

Images