Automatic weld joint alignment tolerance detection method fusing point cloud and image segmentation

Abstract

The invention relates to the technical field of weld joint alignment tolerance detection of special equipment, spaceflight, bridges, ships, wind power and the like, such as containers, pipelines, boilers, recreation facilities, hoisting machinery, passenger ropeways and the like. The invention discloses an automatic weld joint alignment tolerance detection method fusing point cloud and image segmentation, which can quickly and accurately measure the alignment tolerance of a weld joint by combining binocular stereoscopic vision, AI and point cloud analysis technologies, and make up for the defects of the traditional detection method in efficiency and accuracy. The method comprises the following specific steps: shooting through a binocular structured light camera to obtain a grey-scale image and a point cloud image which are aligned; segmenting a welding seam on the grey-scale map, generating a position area mask according to the segmented welding seam, and dividing the actual object into an upper welding seam area, a welding seam area and a lower welding seam area; and according to the segmented welding seam, a welding seam boundary mask is extracted, the position mask and the boundary mask are mapped to a point cloud picture, and regional point cloud data and boundaries are obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of weld misalignment detection for special equipment, aerospace, bridges, ships, wind power, etc., such as containers, pipelines, boilers, amusement facilities, cranes, passenger ropeways, etc. Specifically, it is an automatic weld misalignment detection method that integrates point cloud and image segmentation. Background Art

[0002] Welding technology is widely used in fields such as special equipment, aerospace, bridges, ships, wind power, etc., such as containers, pipelines, boilers, amusement facilities, cranes, passenger ropeways, etc. The weld quality of these devices directly affects their safety and reliability. As an important indicator for evaluating the forming quality of welds, the misalignment amount can reflect the misalignment amount of two adjacent welded parts at the joint, thereby affecting the structural strength and service life of the device.

[0003] When measuring the misalignment amount, the existing methods (as shown in Figure 1 ) generally first place the main scale against one side of the weld, and then slide the height gauge to make it contact the other side of the weld. When the height gauge contacts the other side of the welded part, the reading of the height gauge is the misalignment amount. The existing methods have the following problems: The measurement by the weld gauge relies heavily on the experience of the operator. It is necessary to visually locate the position where the maximum misalignment amount exists, which is time-consuming and laborious, and the detection efficiency is low. Different inspectors may obtain different detection results, with strong subjectivity; The weld gauge can only measure the misalignment amount value of one point at a time.

[0004] Therefore, there is an urgent need for an automatic weld misalignment detection method that can integrate point cloud and image segmentation technologies to improve the detection accuracy and efficiency, reduce costs, and meet the actual needs in the manufacturing of special equipment. Summary of the Invention

[0005] The purpose of the present invention is to provide an automatic weld misalignment detection method that integrates point cloud and image segmentation. This method can quickly and accurately measure the misalignment amount of the weld by combining binocular stereo vision, AI analysis, and point cloud analysis technologies, making up for the deficiencies of traditional detection methods in terms of efficiency and accuracy, and providing a reliable technical means for actual detection.

[0006] The present invention is realized through the following technical solutions: An automatic weld misalignment detection method that integrates point cloud and image segmentation, including the following specific steps: 1) Photograph the object to be measured with a binocular structured light camera, and use binocular stereo vision technology to obtain the aligned grayscale image and point cloud image; 2) Perform target segmentation on the grayscale image, find the weld in the grayscale image, and segment the weld from the background; 3) Generate a position area mask based on the segmented weld, and divide the actual object area into the area above the weld, the weld area, and the area below the weld; 4) Obtain the boundary mask of the weld seam based on the segmented weld seam positions: 5) Map the position area mask and the boundary mask into the point cloud map, and obtain the point cloud data and the point cloud boundary of the divided areas; wherein, the point cloud data includes the upper area point cloud of the weld seam, the weld seam point cloud, and the lower area point cloud of the weld seam, and the point cloud boundary includes the upper boundary point cloud and the lower boundary point cloud; 6) Perform bounding box clipping on all the obtained point cloud data, filter the point cloud data by setting the point cloud boundary threshold, and then perform noise filtering based on the Mahalanobis Distance to remove the noise points in the point cloud; 7) Perform principal component analysis (PCA) on the filtered weld seam point cloud to determine the main axis direction of the weld seam point cloud; calculate the offset matrix between this main axis and the X-axis, and apply this offset matrix to the transformation of the upper area point cloud of the weld seam, the lower area point cloud of the weld seam, the upper boundary point cloud, and the lower boundary point cloud; 8) Adopt a surface fitting method that combines RANSAC and the least squares method to perform surface fitting on the lower area point cloud of the weld seam to obtain the surface equation of the lower area of the weld seam, perform surface completion and extension along the direction from the lower boundary point cloud to the weld seam area, and calculate all the height values Z_fit of all points' X and Y coordinates on the extended surface using the X and Y coordinates of all points in the upper boundary point cloud; 9) According to the calculated height value Z_fit and the height value Z of the initial upper boundary point cloud, calculate the height difference Z_diff between the corresponding points, and take the maximum value of Z_diff as the result of the maximum misalignment amount between the upper and lower area point clouds of the weld seam.

[0007] Furthermore, to better implement an automatic detection method for the misalignment amount of a weld seam that combines point cloud and image segmentation described in the present invention, the following setting method is particularly adopted: Step 1) includes the following specific steps: 1.1) Take pictures of the object to be measured through the left view and the right view of the binocular structured light camera to obtain the left grayscale image and the right grayscale image respectively. The left grayscale image and the right grayscale image together form the grayscale image; 1.2) Project projection speckle structured light onto the left grayscale image and the right grayscale image to generate the left grayscale image and the right grayscale image with speckle structured light; 1.3) After step 1.2), perform feature point matching on the left grayscale image and the right grayscale image, and use the binocular stereo matching technology to generate the point cloud map of the object to be measured.

[0008] To further better implement the automatic weld misalignment detection method integrating point cloud and image segmentation of the present invention, the following setting methods are particularly adopted: In step 2), when performing weld segmentation, it is completed using the fine-tuned target segmentation network FastSAM, and the weld segmentation region and the minimum bounding box obtained based on the weld segmentation region are obtained.

[0009] To further better implement the automatic weld misalignment detection method integrating point cloud and image segmentation of the present invention, the following setting methods are particularly adopted: Step 3) includes the following steps: 3.1) Extend the upper and lower boundaries of the minimum bounding box to the boundaries of the grayscale image at both left and right ends as the new upper and lower boundary lines; 3.2) Define the region between the new upper and lower boundary lines as the weld region, the region above the new upper boundary line as the upper weld region, and the region below the new lower boundary line as the lower weld region; 3.3) Create a blank image, map the three regions of the weld region, the upper weld region, and the lower weld region into it, and name this image box_mask.

[0010] To further better implement the automatic weld misalignment detection method integrating point cloud and image segmentation of the present invention, the following setting methods are particularly adopted: Step 4) includes the following specific steps: 4.1) Extend the upper and lower boundaries of the minimum bounding box to the boundaries of the grayscale image at both left and right ends as the new upper and lower boundaries; 4.2) Create a blank image, map the new upper and lower boundaries into the image, and name this image boarder_mask.

[0011] To further better implement the automatic weld misalignment detection method integrating point cloud and image segmentation of the present invention, the following setting methods are particularly adopted: Step 5) includes the following specific steps: 5.1) According to box_mask and the corresponding point cloud image, map the upper weld region, the weld region, and the lower weld region in box_mask into the point cloud image to obtain the upper weld region point cloud, the weld point cloud, and the lower weld region point cloud; 5.2) According to boarder_mask and the corresponding point cloud image, map the upper boundary and the lower boundary in boarder_mask into the point cloud image to obtain the upper boundary point cloud and the lower boundary point cloud.

[0012] To further better implement the automatic weld misalignment detection method integrating point cloud and image segmentation of the present invention, the following setting methods are particularly adopted: Step 6) includes the following steps: 6.1) Perform a bounding box clipping operation on the three types of point cloud data of the upper weld area point cloud, the lower weld area point cloud, and the weld point cloud obtained; the specific bounding box clipping operation is as follows: Screen the point cloud data (upper weld area point cloud, lower weld area point cloud, weld point cloud) through a preset point cloud boundary threshold to eliminate invalid points or abnormal points outside the set spatial range, thereby initially optimizing the spatial distribution of the point cloud data set; 6.2) After step 6.1), perform noise filtering based on the Mahalanobis Distance on the initially filtered upper weld area point cloud, lower weld area point cloud, and weld point cloud respectively. By calculating the statistical distance of each point relative to the distribution of the point cloud data, identify and remove the noise points deviating from the main data distribution; specifically, the Mahalanobis Distance comprehensively considers the covariance structure of the point cloud data and can effectively distinguish normal points from abnormal noise points, thereby further improving the quality and accuracy of the point cloud data.

[0013] To better implement the automatic detection method for weld misalignment amount integrating point cloud and image segmentation of the present invention, the following setting method is particularly adopted: Step 7) includes the following specific steps: 7.1) Perform principal component analysis (PCA) on the filtered weld point cloud to calculate and determine the main axis direction of the weld point cloud; 7.2) After step 7.1), calculate the offset matrix between the main axis and the X-axis, and apply the offset matrix to the upper weld area point cloud, the lower weld area point cloud, the upper boundary point cloud, and the lower boundary point cloud to complete the rotation alignment of the point cloud.

[0014] To better implement the automatic detection method for weld misalignment amount integrating point cloud and image segmentation of the present invention, the following setting method is particularly adopted: The grayscale image includes the left grayscale image obtained from the left view of the binocular structured light camera and the right grayscale image obtained from the right view of the binocular structured light camera; The point cloud image can also be obtained by using binocular vision stereo imaging technology, multi-view reconstruction, monocular vision stereo imaging technology, or NeRF technology, etc. for the grayscale image, and during the process of reconstructing the grayscale image into a point cloud image, a 2D image to 3D point cloud image alignment operation (i.e., C2P alignment) is also performed.

[0015] Compared with the prior art, the present invention has the following advantages and beneficial effects: In the pressure vessel inspection industry, the present invention can be applied to the measurement and quality assessment of weld misalignment amount, and is suitable for quality control during the manufacturing process and rapid detection during regular inspections, ensuring the safety and reliability of equipment operation.

[0016] In boiler inspection, the present invention can be used for measuring the offset amount and defect assessment of boiler welds and the surfaces of pressure-bearing components, helping to improve the safety and stability of equipment operation.

[0017] In hoisting machinery inspection, the present invention can be applied to the offset measurement and defect detection of pressure-bearing welds with a cylindrical structure, such as the cylindrical columns of portal cranes, the flexible legs of shipbuilding gantry cranes, the cylindrical legs and cylindrical tie rods of container gantry cranes, etc. By detecting the offset amount on the surface of the cylinder, it is ensured that the above cylindrical structures do not undergo structural deformation or cracking during use, effectively guaranteeing the structural stability during service.

[0018] In amusement facility inspection, the present invention can be applied to the offset measurement and defect detection of pressure-bearing welds with a cylindrical structure, such as the cylindrical columns in Ferris wheels, the cylindrical tie rods of space shuttles, and the pressure-bearing tracks with a cylindrical structure, etc. By detecting the offset amount on the surface of the cylinder, it can help achieve the control of manufacturing and assembly accuracy and the dynamic assessment of fatigue damage of key welds during operation, ensuring the safety and reliability of the equipment throughout its life cycle.

[0019] In the inspection of metal and non-metal pipelines, the present invention can be applied to the offset amount detection of welds of pipelines made of different materials, helping to identify the risk of seal failure of high-temperature and high-pressure metal pipelines and the structural deformation caused by misalignment in the installation of non-metal flexible pipelines, ensuring the integrity and operation reliability of the pipeline system.

[0020] In the aerospace field, the present invention can be applied to the offset amount detection of welds of key components such as spacecraft pressure vessels and fuel storage tanks, helping to identify weld defects caused by manufacturing or assembly, ensuring the structural integrity and sealing performance of welds under high temperature, high pressure and extreme environments, and guaranteeing the safety and mission reliability of spacecraft launch and operation.

[0021] By combining binocular stereo vision, AI analysis and point cloud analysis technologies, the present invention can quickly and accurately measure the offset amount of welds, making up for the deficiencies of traditional inspection methods in terms of efficiency and accuracy, and providing a reliable technical means for actual inspection.

[0022] The present invention proposes an automatic analysis method for offset amount based on 3D scanned point cloud, promoting the offset amount detection towards digital detection.

[0023] The present invention combines 2D and 3D technologies, accelerating the analysis speed, which can be carried out on the edge deployment board, making the analysis lightweight.

[0024] The present invention proposes an automatic analysis method for offset amount based on 3D point cloud, reducing the dependence on operators, improving the repeatability and accuracy of detection, and automatically measuring the offset amounts at various locations on the weld with one scan.

[0025] The accuracy of the point cloud formed by the binocular structured light three-dimensional imaging technology can reach an accuracy of 0.01 mm. By directly analyzing the scanned surface, the analysis accuracy of the present invention is greatly improved.

[0026] The present invention can calculate all regions of the weld, not just give the values of one region or one line.

[0027] The present invention is carried out in the form of combining 2D + 3D, and the algorithm can be deployed on lightweight edge devices.

[0028] The algorithm of the present invention combines 2D image recognition and 3D point cloud analysis, resulting in high recognition accuracy and high repeatability. Description of the Drawings

[0029] Figure 1 Schematic diagram of the misalignment measurement of the prior art.

[0030] Figure 2 Actual image of the object to be measured obtained from the left view of the binocular structured light camera.

[0031] Figure 3 Point cloud map of the object generated after feature point matching on the grayscale image using the binocular stereo matching technology.

[0032] Figure 4 Result display diagram of weld segmentation on the grayscale image.

[0033] Figure 5 Position area mask map generated by the segmented weld.

[0034] Figure 6 Boundary mask map of the weld.

[0035] Figure 7 Map formed by mapping the position area mask to the point cloud map.

[0036] Figure 8 Map formed by mapping the boundary mask to the point cloud map.

[0037] Figure 9 Result diagram after point cloud rotation alignment.

[0038] Figure 10 Schematic diagram of all height values Z_fit on the extended surface. Detailed Description of the Invention

[0039] The present invention will be further described in detail below in conjunction with embodiments, but the embodiments of the present invention are not limited thereto.

[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0041] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "a plurality" means two or more unless otherwise specifically defined.

[0042] Example 1: The present invention designs an automatic detection method for weld misalignment amount that integrates point cloud and image segmentation. This method combines binocular stereo vision, AI analysis, and point cloud analysis technologies to quickly and accurately measure the weld misalignment amount, making up for the deficiencies of traditional detection methods in terms of efficiency and accuracy, and providing a reliable technical means for actual detection. The specific steps are as follows: 1) Photograph the object to be measured with a binocular structured light camera, and use binocular stereo vision technology to obtain the aligned grayscale image and point cloud image; 2) Perform target segmentation on the grayscale image to find the weld in the grayscale image and segment the weld from the background; 3) Generate a position area mask based on the segmented weld, and divide the actual object area into the upper area of the weld, the weld area, and the lower area of the weld; 4) Obtain the boundary mask of the weld according to the segmented weld position: 5) Map the position area mask and the boundary mask to the point cloud image, and obtain the sub-region point cloud data and the point cloud boundary; among them, the point cloud data includes the upper area point cloud of the weld, the weld point cloud, and the lower area point cloud of the weld, and the point cloud boundary includes the upper boundary point cloud and the lower boundary point cloud; 6) Perform bounding box clipping on all the acquired point cloud data, filter the point cloud data by setting the point cloud boundary threshold, and then perform noise filtering based on the Mahalanobis Distance to remove the noise points in the point cloud; 7) Perform principal component analysis on the filtered weld point cloud to determine the main axis direction of the weld point cloud; calculate the offset matrix between this main axis and the X-axis, and apply this offset matrix to the transformation of the point cloud in the upper weld area, the point cloud in the lower weld area, the upper boundary point cloud, and the lower boundary point cloud; 8) Adopt a surface fitting method that combines RANSAC and the Least Squares Method (MLS) to perform surface fitting on the point cloud in the lower weld area to obtain the surface equation of the lower weld area, perform surface completion and extension along the direction from the lower boundary point cloud to the weld area, and use the X and Y coordinates of all points in the upper boundary point cloud to calculate all the height values Z_fit of the X and Y coordinates of all points on the extended surface; 9) According to the calculated height value Z_fit and the height value Z of the initial upper boundary point cloud, calculate the height difference Z_diff between the corresponding points, and take the maximum value of Z_diff as the result of the maximum misalignment amount between the point clouds in the upper and lower weld areas.

[0043] Example 2: This example is further optimized on the basis of the above example. The same parts as the foregoing technical solutions will not be elaborated here. To better implement an automatic detection method for weld misalignment amount that combines point cloud and image segmentation of the present invention, the following specific settings are particularly adopted: The step 1) includes the following specific steps: 1.1) Take pictures of the object to be measured through the left view and the right view of the binocular structured light camera to obtain the left grayscale image and the right grayscale image respectively. The left grayscale image and the right grayscale image together form the grayscale image; 1.2) Project projection speckle structured light on the left grayscale image and the right grayscale image to generate the left grayscale image and the right grayscale image with speckle structured light; 1.3) After step 1.2), perform feature point matching on the left grayscale image and the right grayscale image, and use binocular stereo matching technology to generate the point cloud map of the object to be measured.

[0044] Example 3: This example is further optimized on the basis of any of the above examples. The same parts as the foregoing technical solutions will not be elaborated here. To better implement an automatic detection method for weld misalignment amount that combines point cloud and image segmentation of the present invention, the following specific settings are particularly adopted: When performing weld segmentation in the step 2), it is completed using the fine-tuned target segmentation network FastSAM, and the weld segmentation area and the minimum bounding box obtained according to the weld segmentation area are obtained.

[0045] Example 4: This embodiment is further optimized on the basis of any of the above embodiments. The same parts as the foregoing technical solutions will not be described herein again. To better implement an automatic detection method for weld misalignment amount that fuses point cloud and image segmentation, the following setting method is particularly adopted: Step 3) includes the following steps: 3.1) Extend the upper and lower boundaries of the minimum bounding box to the boundaries of the grayscale image at both left and right ends as the new upper and lower boundary lines; 3.2) Define the area between the new upper and lower boundary lines as the weld area, the area above the new upper boundary line as the upper weld area, and the area below the new lower boundary line as the lower weld area; 3.3) Create a blank image, map the three areas of the weld area, the upper weld area, and the lower weld area into it, and name this image box_mask.

[0046] Example 5: This embodiment is further optimized on the basis of any of the above embodiments. The same parts as the foregoing technical solutions will not be described herein again. To better implement an automatic detection method for weld misalignment amount that fuses point cloud and image segmentation, the following setting method is particularly adopted: Step 4) includes the following specific steps: 4.1) Extend the upper and lower boundaries of the minimum bounding box to the boundaries of the grayscale image at both left and right ends as the new upper and lower boundaries; 4.2) Create a blank image, map the new upper and lower boundaries into the image, and name this image boarder_mask.

[0047] Example 6: This embodiment is further optimized on the basis of any of the above embodiments. The same parts as the foregoing technical solutions will not be described herein again. To better implement an automatic detection method for weld misalignment amount that fuses point cloud and image segmentation, the following setting method is particularly adopted: Step 5) includes the following specific steps: 5.1) According to box_mask and the corresponding point cloud image, map the upper weld area, the weld area, and the lower weld area in box_mask into the point cloud image to obtain the upper weld area point cloud, the weld point cloud, and the lower weld area point cloud; 5.2) According to boarder_mask and the corresponding point cloud image, map the upper boundary and the lower boundary in boarder_mask into the point cloud image to obtain the upper boundary point cloud and the lower boundary point cloud.

[0048] Example 7: This embodiment is further optimized on the basis of any of the above embodiments. The same parts as the foregoing technical solutions will not be described herein again. To better implement an automatic detection method for weld misalignment amount that fuses point cloud and image segmentation, the following setting method is specifically adopted: Step 6) includes the following steps: 6.1) Perform a bounding box clipping operation on the three types of point cloud data of the upper region point cloud, lower region point cloud, and weld point cloud of the weld obtained; the bounding box clipping operation is specifically: filter the point cloud data (upper region point cloud, lower region point cloud, weld point cloud of the weld) through a preset point cloud boundary threshold to remove invalid points or abnormal points outside the set spatial range, so as to initially optimize the spatial distribution of the point cloud data set; 6.2) After step 6.1), perform noise filtering based on the Mahalanobis Distance on the preliminarily filtered upper region point cloud, lower region point cloud, and weld point cloud of the weld respectively. By calculating the statistical distance of each point relative to the distribution of the point cloud data, identify and remove the noise points deviating from the main data distribution; specifically, the Mahalanobis Distance comprehensively considers the covariance structure of the point cloud data and can effectively distinguish normal points from abnormal noise points, thereby further improving the quality and accuracy of the point cloud data.

[0049] Example 8: This embodiment is further optimized on the basis of any of the above embodiments. The same parts as the foregoing technical solutions will not be described herein again. To better implement an automatic detection method for weld misalignment amount that fuses point cloud and image segmentation, the following setting method is specifically adopted: Step 7) includes the following specific steps: 7.1) Perform a principal component analysis (PCA) on the filtered weld point cloud to calculate and determine the principal axis direction of the weld point cloud; 7.2) After step 7.1), calculate the offset matrix between the principal axis and the X-axis, and apply the offset matrix to the upper region point cloud, lower region point cloud, upper boundary point cloud, and lower boundary point cloud of the weld to rotate and align the point cloud.

[0050] Example 9: This embodiment is further optimized on the basis of any of the above embodiments. The same parts as the foregoing technical solutions will not be described herein again. To better implement an automatic detection method for weld misalignment amount that fuses point cloud and image segmentation, the following settings are particularly adopted: The grayscale image includes a left grayscale image obtained from the left view of a binocular structured light camera and a right grayscale image obtained from the right view of the binocular structured light camera; The point cloud image can also be obtained by using binocular vision stereoscopic imaging technology, multi-view reconstruction, monocular vision stereoscopic imaging technology, or NeRF technology, etc. on the grayscale image. And during the process of reconstructing the grayscale image into a point cloud image, an alignment operation from 2D image to 3D point cloud image (i.e., C2P alignment) is also performed.

[0051] Embodiment 10: An automatic detection method for weld misalignment amount that fuses point cloud and image segmentation, comprising the following steps: First step, photograph the object to be measured through the left view and the right view of a binocular structured light camera, and respectively obtain a left grayscale image (as Figure 2 shown) and a right grayscale image; project projection speckle structured light on the left grayscale image and the right grayscale image to generate a left grayscale image and a right grayscale image with speckle structured light; then, perform feature point matching on the left grayscale image and the right grayscale image, and use binocular stereo matching technology to generate a point cloud image of the object to be measured. The left grayscale image obtained from the left view of the binocular structured light camera and the right grayscale image obtained from the right view of the binocular structured light camera together constitute the grayscale image. The point cloud image can also be obtained by using binocular vision stereoscopic imaging technology, multi-view reconstruction, monocular vision stereoscopic imaging technology, or NeRF technology, etc. on the grayscale image. During the process of reconstructing the grayscale image into a point cloud image, an alignment operation from 2D image to 3D point cloud image (i.e., C2P alignment) is also performed, and the point cloud image is as Figure 3 shown.

[0052] Second step, use the fine-tuned target segmentation network FastSAM to perform target segmentation on the grayscale image (2D image), find the weld in the grayscale image, and segment the weld from the background to obtain the weld segmentation region and the minimum bounding box obtained according to the weld segmentation region. The segmentation result is as Figure 4 shown. In this figure, the red segmentation region is the segmentation region obtained by the algorithm, that is, the region where the weld is located, and the circumscribed red box is the actual minimum bounding box obtained according to the segmentation region.

[0053] Third step, according to the minimum bounding box obtained in the second step, divide the 2D grayscale image into three regions: taking Figure 4The upper and lower boundaries of the red box in Figure 5 extend to the picture boundaries at both the left and right ends, serving as the new upper and lower boundary lines; the area between the new upper and lower boundary lines is defined as the weld area, the area above the new upper boundary line is the upper weld area, and the area below the new lower boundary line is the lower weld area. Create a blank picture and map the above three areas into it. As

[0054] shown, where the upper weld area is mapped to black, the weld area is mapped to gray, and the lower weld area is mapped to white, and name this picture box_mask. Figure 4 In Figure 6 the upper and lower boundaries of the red box in

[0055] extend to the picture boundaries at both the left and right ends, serving as the new upper and lower boundaries. Create a blank picture and map the new upper and lower boundaries into the picture. As Figure 7 shown, where the pure white color represents the upper boundary and the off-white color represents the lower boundary, and name this picture boarder_mask.

[0056] Step 5: According to box_mask and the corresponding point cloud picture, map the upper weld area, weld area, and lower weld area in box_mask into the point cloud picture to obtain the upper weld area point cloud, weld point cloud, and lower weld area point cloud. As Figure 7 shown, where the blue point set is the upper weld area point set, the green area is the weld click, and the red area is the lower weld area point set.

[0056] Step 6: According to boarder_mask and the corresponding point cloud picture, map the upper boundary and lower boundary in boarder_mask into the point cloud picture to obtain the upper boundary point cloud and lower boundary point cloud. As Figure 8 shown, where the green point set is the upper boundary point cloud and the red point set is the lower boundary point cloud.

[0057] Step 7: Perform a bounding box clipping operation on the upper weld area point cloud, lower weld area point cloud, and weld point cloud obtained in Step 5; the specific bounding box clipping operation is: screen the point cloud data (upper weld area point cloud, lower weld area point cloud, weld point cloud) through a preset point cloud boundary threshold to eliminate invalid points or abnormal points that exceed the set spatial range, thereby initially optimizing the spatial distribution of the point cloud data set.

[0058] Step 8: Perform noise filtering based on the Mahalanobis Distance on the point clouds of the upper region of the weld, the point clouds of the lower region of the weld, and the point clouds of the weld respectively. By calculating the statistical distance of each point relative to the distribution of the point cloud data, identify and remove the noise points that deviate from the main data distribution. Specifically, the Mahalanobis Distance comprehensively considers the covariance structure of the point cloud data and can effectively distinguish normal points from abnormal noise points, thereby further improving the quality and accuracy of the point cloud data.

[0059] Step 9: Perform principal component analysis (PCA) on the filtered point clouds of the weld, calculate and determine the main axis direction of the point clouds of the weld; then calculate the offset matrix between the main axis and the X-axis, and apply the offset matrix to the point clouds of the upper region of the weld, the point clouds of the lower region of the weld, the upper boundary point clouds, and the lower boundary point clouds to complete the rotational alignment of the point clouds. The result after rotation is as Figure 9 shown. The blue area is the point set of the upper region of the weld, the red area is the point set of the lower region of the weld, and the green area is the point set of the weld region. In the figure, the red, blue, and green coordinate axes respectively represent the X-axis, Y-axis, and Z-axis in the world coordinate system.

[0060] Step 10: Adopt a surface fitting method that combines RANSAC and the least squares method (MLS) to perform surface fitting on the point clouds of the lower region of the weld. According to the surface equation of the lower region of the weld obtained by fitting, perform surface completion and extension along the direction from the lower boundary point clouds to the weld region, and use the X and Y coordinates of all points in the upper boundary point clouds to calculate all the height values Z_fit of the X and Y coordinates of all points on the extended surface; the result is as Figure 10 shown. Figure 10 It includes a top view (left) and a side view (right). Among them, the red point set is the point clouds of the upper region of the weld, the gray point set is the point clouds of the lower region of the weld, and the green point set is the result of fitting the point clouds of the lower region of the weld and extending along the weld region direction. In the top view, the blue point set represents the upper boundary point clouds; in the side view, the yellow point set is the point set obtained by substituting the X and Y coordinates of the upper boundary point clouds into the surface equation of the lower region of the weld.

[0061] Step 11: According to the calculated height value Z_fit and the height value Z of the initial upper boundary point clouds, calculate the height difference Z_diff between the corresponding points, and take the maximum value of Z_diff as the result of the maximum misalignment amount between the point clouds of the upper and lower regions of the weld.

[0062] The above are only the preferred embodiments of the present invention, and do not impose any form of limitation on the present invention. Any simple modification or equivalent change made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for automatically detecting weld misalignment by integrating point cloud and image segmentation, characterized in that: The specific steps include: 1) Use a binocular structured light camera to shoot the object to be measured, and use binocular stereo vision technology to obtain the aligned grayscale image and point cloud image; 2) Perform target segmentation on the grayscale image, find the weld in the grayscale image, and separate the weld from the background; 3) Generate a position area mask based on the segmented weld, and divide the actual object area into the weld upper area, weld area and weld lower area; 4) Obtain the weld boundary mask based on the segmented weld position: 5) Map the position area mask and boundary mask to the point cloud image, and obtain the point cloud data and point cloud boundaries of the sub-regions; the point cloud data includes the point cloud of the upper area of ​​the weld, the point cloud of the weld, and the point cloud of the lower area of ​​the weld, and the point cloud boundaries include the upper boundary point cloud and the lower boundary point cloud; 6) Perform bounding box cropping on all acquired point cloud data, filter the point cloud data by setting the point cloud boundary threshold, and then perform noise filtering based on the Mahalanobis distance to remove noise points in the point cloud; 7) Perform principal component analysis on the filtered weld point cloud to determine the principal axis direction of the weld point cloud; calculate the offset matrix between the principal axis and the X-axis, and apply this offset matrix to the transformation of the weld upper area point cloud, the weld lower area point cloud, the upper boundary point cloud, and the lower boundary point cloud; 8) The surface fitting method integrating RANSAC and least squares method is used to fit the point cloud of the area under the weld and obtain the surface equation of the area under the weld. The surface is completed and extended along the direction of the lower boundary point cloud pointing to the weld area. The X and Y coordinates of all points in the upper boundary point cloud are used to calculate all the height values ​​Z_fit of the X and Y coordinates of all points on the extended surface. 9) According to the calculated height value Z_fit and the height value Z of the initial upper boundary point cloud, calculate the height difference Z_diff between the corresponding points, and take the maximum value of Z_diff as the maximum misalignment result between the point clouds in the upper and lower areas of the weld.

2. According to the method for automatic detection of weld misalignment by integrating point cloud and image segmentation in claim 1, it is characterized by: The step 1) includes the following specific steps: 1.1) The object to be measured is photographed through the left and right perspectives of the binocular structured light camera to obtain a left grayscale image and a right grayscale image respectively, and the left grayscale image and the right grayscale image constitute a grayscale image as a whole; 1.2) Projecting speckle structured light onto the left grayscale image and the right grayscale image to generate a left grayscale image and a right grayscale image with speckle structured light; 1.3) After step 1.2), feature point matching is performed on the left grayscale image and the right grayscale image, and a point cloud image of the object being measured is generated using binocular stereo matching technology.

3. The automatic detection method of weld misalignment by integrating point cloud and image segmentation according to claim 1 is characterized in that: In the step 2), when performing weld segmentation, the target segmentation network FastSAM is used after fine-tuning to complete the segmentation, and the weld segmentation area and the minimum closure box obtained according to the weld segmentation area are obtained.

4. The method for automatically detecting weld misalignment by integrating point cloud and image segmentation according to claim 3 is characterized in that: The step 3) comprises the following steps: 3.1) Extend the upper and lower boundaries of the minimum enclosing box to the left and right ends to the grayscale image boundary as the new upper and lower boundary lines; 3.2) The area between the new upper and lower boundary lines is defined as the weld area, the area above the new upper boundary line is the upper weld area, and the area below the new lower boundary line is the lower weld area; 3.3) Create a blank image, map the weld area, the upper weld area, and the lower weld area into it, and name the image box_mask.

5. The method for automatically detecting weld misalignment by integrating point cloud and image segmentation according to claim 4, characterized in that: The step 4) includes the following specific steps: 4.1) Extend the upper and lower boundaries of the minimum enclosing box to the left and right ends to the grayscale image boundary as the new upper and lower boundaries; 4.2) Create a blank map, map the new upper and lower boundaries to the map, and name the map boarder_mask.

6. The method for automatically detecting weld misalignment by integrating point cloud and image segmentation according to claim 5, characterized in that: The step 5) includes the following specific steps: 5.1) According to the box_mask and the corresponding point cloud image, the upper area of ​​the weld, the weld area, and the lower area of ​​the weld in the box_mask are mapped to the point cloud image to obtain the point cloud of the upper area of ​​the weld, the weld point cloud, and the point cloud of the lower area of ​​the weld; 5.2) According to the boarder_mask and the corresponding point cloud image, the upper boundary and the lower boundary in the boarder_mask are mapped to the point cloud image to obtain the upper boundary point cloud and the lower boundary point cloud.

7. The method for automatically detecting weld misalignment by integrating point cloud and image segmentation according to claim 6, characterized in that: The step 6) comprises the following steps: 6.1) Performing a bounding box clipping operation on the three types of point cloud data obtained, namely, the point cloud of the upper area of ​​the weld, the point cloud of the lower area of ​​the weld, and the weld point cloud; the bounding box clipping operation specifically includes: filtering the point cloud data by a preset point cloud boundary threshold to remove invalid points or abnormal points that exceed the set spatial range, thereby preliminarily optimizing the spatial distribution of the point cloud data set; 6.2) After step 6.1), the initially filtered weld upper area point cloud, weld lower area point cloud, and weld point cloud are respectively subjected to noise filtering based on the Mahalanobis distance. By calculating the statistical distance of each point relative to the point cloud data distribution, the noise points that deviate from the main distribution of the data are identified and removed.

8. The method for automatically detecting weld misalignment by integrating point cloud and image segmentation according to claim 1, characterized in that: The step 7) includes the following specific steps: 7.1) Perform principal component analysis on the filtered weld point cloud and calculate and determine the principal axis direction of the weld point cloud; 7.2) After step 7.1), calculate the offset matrix between the main axis and the X axis, and apply the offset matrix to the point cloud of the upper area of ​​the weld, the point cloud of the lower area of ​​the weld, the upper boundary point cloud, and the lower boundary point cloud to complete the rotational alignment of the point cloud.

Images