Cluster optimization processing method of weld point cloud and weld track generation method

Abstract

This invention relates to a clustering optimization method for weld point clouds and a weld trajectory generation method, which solves the technical problem of how to more accurately extract weld point clouds from workpiece point cloud information and generate more accurate weld spatial trajectories. The method involves projecting the workpiece point cloud and weld point cloud into two dimensions along preset multi-view directions to generate two-dimensional projected images at corresponding viewpoints; performing instance segmentation processing on the two-dimensional projected images to obtain the segmented regions and category information of the weld in the two-dimensional view; based on the instance segmentation results, constructing the correspondence between the two-dimensional segmented regions and the three-dimensional weld points to achieve clustering of the three-dimensional weld points; finally, for different weld point clusters, performing geometric configuration fitting according to their corresponding weld categories to obtain the spatial trajectory of the weld.

Description

Technical Field

[0001] This invention relates to the field of flexible autonomous intelligent welding technology, and more specifically, to a method for clustering and optimizing weld point clouds and a method for generating weld trajectories. Background Technology

[0002] Flexible, autonomous, and intelligent welding in the shipbuilding and steel structure industries requires the use of visual algorithms to detect the weld position. After acquiring basic point cloud information and weld trajectory data of the workpiece, the welding torch is automatically moved to the weld's starting position and adjusted to the correct posture to begin welding. Currently, welding is mainly performed using a teaching method. However, for non-standard weldments, or in situations with large placement errors, or numerous interference factors at the arc welding site, the teaching-based welding operation lacks sufficient intelligence and is inefficient.

[0003] For non-standard weldments, weldments without drawings, or weldments with large placement errors, teaching methods have limitations. Therefore, instead of teaching programming, deep learning algorithms are used to identify and locate weld seams to achieve intelligent welding. The basic process of using deep learning for weld seam identification can be found in the invention patent application CN110110798A, entitled "A Weld Seam Recognition Method Based on Mask-RCNN Network". However, the aforementioned deep learning-based weld seam recognition algorithm is a method for target recognition based on two-dimensional images. The detection results lack depth information, therefore the recognition accuracy is not high.

[0004] Another method for weld seam identification based on point clouds of welded parts, referring to the invention patent application CN119107306A entitled "Model-free Weld Seam Identification Method and Apparatus," identifies weld seams from the point cloud information of the workpiece. However, the accuracy of this method in identifying weld seams still needs improvement.

[0005] Weld seam recognition methods based on 3D point clouds can directly utilize the depth information and real 3D geometric features of the weldment, exhibiting stronger robustness in complex weldment structures, occluded scenes, and under varying lighting conditions. By collecting point cloud information from the workpiece, weld seam point clouds are extracted, and a fitting method is selected based on these point clouds to generate the weld seam trajectory. In conclusion, how to more accurately extract weld seam point clouds and generate more precise weld seam spatial trajectories is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] The present invention aims to solve the technical problem of how to extract weld point clouds more accurately from the point cloud information of workpieces and generate more accurate weld spatial trajectories, and provides a clustering optimization processing method for weld point clouds and a weld trajectory generation method.

[0007] A first aspect of the present invention provides a method for clustering optimization of weld seam point clouds, comprising the following steps: Step (1): Establish multiple projection field ranges based on orthogonal projection perspectives; Step (2): The weld point cloud and the weld point cloud including multiple weld seam segments are orthophoto-projected onto the projection field of view, and pixels are formed in the projection field of view to obtain a two-dimensional projection image; since there are multiple projection field of view, multiple two-dimensional projection images are obtained. Step (3): Color the multiple two-dimensional projection images. Use a first color to color the pixels obtained from the weldment point cloud projection, and use a second color that is significantly different from the first color to color the pixels obtained from the weld seam point cloud projection, to obtain the two-dimensional projection image after color rendering. ; Step (4) Render the two-dimensional projection image after coloring and drawing. Input the YOLOv8-Seg weld seam recognition instance segmentation model, and generate a 2D projection image of the YOLOv8-Seg weld seam recognition instance segmentation model after color rendering. Perform instance segmentation and output the instance segmentation result image. The instance segmentation result image Label containing multiple polygon instance separators k i Label k i ={Label k 1 Label k 2 Label k 3 ,…,Label k n}, where n represents the number of polygon instance bounding boxes, and Label k i Includes the weld segment category information TYPE corresponding to the i-th polygon instance segmentation box; Step (5): Based on the instance segmentation result image The weld point cloud containing multiple weld segments is divided into multiple point cloud clusters containing only one weld segment; Step (5-1) segment multiple instance result images Polygon instance divider label k i Perform permutations and combinations to generate instance combinations (P1, P2, ..., PK), where the set of values ​​for P1 is {Label1}. i}, the set of values ​​for P2 is {Label2} i}, the set of values ​​for PK is {Label k i}; Step (5-2) involves clustering the weld point cloud containing multiple weld segments, and then segmenting the points in the weld point cloud into the instance segmentation result image. Up projection, find the segmentation result image that falls into each instance simultaneously. Medium polygonal frame instance box Label k i The points in the cloud are grouped into point cloud clusters. All instance combinations (P1, P2, ..., PK) are traversed. If the TYPE information in any item of (P1, P2, ..., PK) is the first type of weld segment information, then the type of the point cloud cluster is determined to be the first type of weld segment information cluster. If the TYPE information in each item of (P1, P2, ..., PK) is the second type of weld segment information, then the type of the point cloud cluster is determined to be the second type of weld segment information cluster.

[0008] Preferably, the first type of weld segment category information is a curve, and the second type of weld segment category information is a straight line.

[0009] Preferably, step (1), which establishes multiple projection field ranges based on orthogonal projection perspectives, is performed in the following manner: Step 1): Calculation of principal axis direction and construction of coordinate system; The point cloud of the weldment is fitted with a random sampling consensus algorithm to obtain a principal plane. The plane normal vector is extracted based on the principal plane and set as the Z-axis direction of the weldment's oriented bounding box. From the X-axis direction vector and Y-axis direction vector of the world coordinate system where the weldment point cloud is located, select one that is not parallel to the plane normal vector as a preset reference direction vector; project this preset reference direction vector onto the main plane to form a projection vector in the main plane, and use this projection vector as the X-axis direction vector with the main plane as the reference. Then, obtain the Y-axis direction vector by performing a cross product operation between the plane normal vector and the X-axis direction vector, and construct a three-dimensional orthogonal coordinate system with the main plane as the reference. The interior points of the principal plane are projected onto the XY plane in the three-dimensional orthogonal coordinate system based on the principal plane to form projection points. Then, principal component analysis is performed on all projection points in the XY plane. The direction corresponding to the first principal component is taken as the updated X-axis direction. Based on the cross product relationship between the updated X-axis direction and the Z-axis direction, the Y-axis direction is corrected to obtain a new principal axis coordinate system. Step 2), calculate the transformation matrix from the world coordinate system to the new principal axis coordinate system, and transform the point cloud of the weldment to the new principal axis coordinate system using the transformation matrix; Step 3), calculate the axis-aligned bounding box of the weldment point cloud in the new principal axis coordinate system using the projection extremum method; Step 4) Set the center point of the axis-aligned bounding box as the projection reference center, and set the projection viewpoints along each axis of the new principal axis coordinate system to form multiple orthogonal projection viewpoints; for each projection viewpoint, create a camera coordinate system with the projection reference center as the camera target point and the projection viewpoint direction as the optical axis direction. The distance between the camera position and the projection reference center is several times the length of the diagonal of the axis-aligned bounding box, and create multiple camera coordinate systems. Step 5) Establish multiple projection field ranges based on each camera coordinate system.

[0010] Preferably, in step (2), the following formula is used: In the formula, T is the translation vector and R is the rotation matrix; The weldment point cloud and the weld point cloud including multiple weld segments are transformed from the new principal axis coordinate system to the camera coordinate system. Then, the weldment point cloud and the weld point cloud including multiple weld segments in the camera coordinate system are orthophotographed onto the projection field of view.

[0011] Preferably, there are three orthogonal projection perspectives: a first principal axis positive direction is formed along the positive X-axis, a second principal axis positive direction is formed along the positive Y-axis, and a third principal axis positive direction is formed along the positive Z-axis.

[0012] A second aspect of the present invention provides a method for clustering and optimizing weld point clouds, comprising the following steps: Step (1): Establish multiple projection field ranges based on orthogonal projection perspectives; Step (2): The point cloud of the weldment and the point cloud of the weld with only one weld segment are orthophotos projected onto the projection field of view, and pixels are formed in the projection field of view to obtain a two-dimensional projection image; since there are multiple projection field of view, multiple two-dimensional projection images are obtained. Step (3): Color the multiple two-dimensional projection images. Use a first color to color the pixels obtained from the weldment point cloud projection, and use a second color that is significantly different from the first color to color the pixels obtained from the weld seam point cloud projection, to obtain the two-dimensional projection image after color rendering. ; Step (4) Render the two-dimensional projection image after coloring and drawing. Input the YOLOv8-Seg weld seam recognition instance segmentation model, and generate a 2D projection image of the YOLOv8-Seg weld seam recognition instance segmentation model after color rendering. Perform instance segmentation and output the instance segmentation result image. The instance segmentation result image It contains a polygon instance segmentation box, and the polygon instance segmentation box contains weld segment category information; Step (5) involves clustering the weld point cloud with only one weld segment, and then segmenting the points in the weld point cloud into the instance segmentation result image. Project upwards, find the points that fall within the polygonal bounding box of each instance segmentation result image, and group these points into point cloud clusters.

[0013] Preferably, step (1), which establishes multiple projection field ranges based on orthogonal projection perspectives, is performed in the following manner: Step 1): Calculation of principal axis direction and construction of coordinate system; The point cloud of the weldment is fitted with a random sampling consensus algorithm to obtain a principal plane. The plane normal vector is extracted based on the principal plane and set as the Z-axis direction of the weldment's oriented bounding box. From the X-axis direction vector and Y-axis direction vector of the world coordinate system where the weldment point cloud is located, select one that is not parallel to the plane normal vector as a preset reference direction vector; project this preset reference direction vector onto the main plane to form a projection vector in the main plane, and use this projection vector as the X-axis direction vector with the main plane as the reference. Then, obtain the Y-axis direction vector by performing a cross product operation between the plane normal vector and the X-axis direction vector, and construct a three-dimensional orthogonal coordinate system with the main plane as the reference. The interior points of the principal plane are projected onto the XY plane in the three-dimensional orthogonal coordinate system based on the principal plane to form projection points. Then, principal component analysis is performed on all projection points in the XY plane. The direction corresponding to the first principal component is taken as the updated X-axis direction. Based on the cross product relationship between the updated X-axis direction and the Z-axis direction, the Y-axis direction is corrected to obtain a new principal axis coordinate system. Step 2), calculate the transformation matrix from the world coordinate system to the new principal axis coordinate system, and transform the point cloud of the weldment to the new principal axis coordinate system using the transformation matrix; Step 3), calculate the axis-aligned bounding box of the weldment point cloud in the new principal axis coordinate system using the projection extremum method; Step 4) Set the center point of the axis-aligned bounding box as the projection reference center, and set the projection viewpoints along each axis of the new principal axis coordinate system to form multiple orthogonal projection viewpoints; for each projection viewpoint, create a camera coordinate system with the projection reference center as the camera target point and the projection viewpoint direction as the optical axis direction. The distance between the camera position and the projection reference center is several times the length of the diagonal of the axis-aligned bounding box, and create multiple camera coordinate systems. Step 5) Establish multiple projection field ranges based on each camera coordinate system.

[0014] Preferably, in step (2), the following formula is used: In the formula, T is the translation vector and R is the rotation matrix; Transform the weldment point cloud and weld seam point cloud from the new principal axis coordinate system to the camera coordinate system, and then orthophoto project the weldment point cloud and weld seam point cloud in the camera coordinate system onto the projection field of view.

[0015] A third aspect of the present invention provides a method for generating a weld trajectory, wherein a weld spatial trajectory is generated based on any of the point cloud clusters described above.

[0016] The technical solution of this invention integrates point cloud geometric features and instance segmentation information. It projects the workpiece point cloud and weld point cloud in two dimensions along preset multi-view directions to generate two-dimensional projected images from the corresponding viewpoints. Instance segmentation processing is then performed on the two-dimensional projected images to obtain the segmented regions and category information of the weld in the two-dimensional view. Based on the instance segmentation results, a correspondence between the two-dimensional segmented regions and the three-dimensional weld points is constructed to achieve clustering of the three-dimensional weld points. Finally, for different weld point clusters, geometric configuration fitting is performed according to their corresponding weld categories to obtain the spatial trajectory of the weld.

[0017] The beneficial effects of this invention are that after optimizing the weld point cloud, a more accurate weld point cluster is obtained, and a more accurate weld spatial trajectory is obtained based on the weld point cluster. The weld spatial trajectory is highly consistent with the actual weld shape, which can meet the trajectory guidance requirements of automated and high-precision welding.

[0018] It is especially suitable for welding complex structural components.

[0019] Further features and aspects of the present invention will be clearly described in the following detailed description with reference to the accompanying drawings. Attached Figure Description

[0020] Figure 1 This is a flowchart for obtaining weld seam point clouds; Figure 2 This is a flowchart for generating weld trajectories based on weld point clouds; Figure 3 This is a schematic diagram showing the connection between each point in the point cloud and its k nearest neighbors, where k is 5. Figure 4 This is a schematic diagram of the diverging point cloud normal vector and the converging point cloud normal vector; Figure 5 It is the point cloud of the extracted straight weld segment; Figure 6 It is the point cloud of the extracted arc-shaped weld segment; Figure 7 This is a schematic diagram of the world coordinate system; Figure 8This is a schematic diagram of the new principal axis coordinate system; Figure 9 This is a structural diagram of the axis-aligned bounding box in the new principal axis coordinate system; Figure 10 It is a structural diagram with three orthogonal projection views; Figure 11 This is a rendering of a point cloud projection from multiple perspectives onto a workpiece. Figure 12 This is a rendering of a point cloud multi-view projection of another type of workpiece; Figure 13 This is a diagram showing the effect of segmenting a two-dimensional projected view into instances. Figure 14 This is a diagram showing the effect of segmenting an instance of a two-dimensional projection view of another workpiece. Figure 15 yes Figure 14 The instance segmentation results show polygon instance segmentation labels. k i ; Figure 16 This is a diagram showing the effect of trajectory fitting for a weld seam on a workpiece. Figure 17 This is a diagram showing the effect of trajectory fitting for a weld seam on another type of workpiece. Figure 18 yes Figure 17 The image shown is from another perspective. Detailed Implementation

[0021] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] The specific embodiments described below are merely preferred embodiments of this application, and the scope of protection of this application is not limited thereto. Those skilled in the art can make modifications or variations based on the principles, concepts, and spirit of this application, and the resulting technical solutions should all be covered within the scope of protection of this application.

[0023] Before welding, the parts to be welded are usually pre-assembled into a whole by spot welding to form a weldment.

[0024] refer to Figure 1 The first step is to acquire the point cloud of the weldment. Specifically, this can be achieved by using a point cloud scanning device (such as a line-scanning binocular camera) to perform a spatial scan of the weldment, collecting the spatial geometric information of its surface, and generating corresponding 3D point cloud data. This 3D point cloud data must include at least the coordinate information of each sampling point on the weldment surface in 3D space.

[0025] The second step is to preprocess the point cloud of the weldment. Specific preprocessing steps may include point cloud denoising, downsampling, coordinate system unification, and / or redundant point removal or region clipping, to improve the stability and accuracy of subsequent weld identification and processing.

[0026] The third step is to estimate the normal vectors of each point in the point cloud of the weldment and then normalize the normal vectors.

[0027] Step (1): Use the K-Nearest Neighbors (KNN) algorithm to process each point in the weldment point cloud. Find k nearest neighbors within its neighborhood. Nearest neighbor points are a set of points. .

[0028] Step (2) uses the PCA (Principal Component Analysis) algorithm to analyze the points. and its nearest neighbors The process yields eigenvalues ​​and their corresponding eigenvectors.

[0029] During the calculation process, based on the k nearest neighbors The average coordinates generate the mean point. Calculate each neighboring point For each offset vector relative to the mean point, perform an outer product operation, and then sum the results of the outer products to obtain the point. and k nearest neighbors The covariance matrix of the point set :

[0030] For covariance matrix Eigenvalue decomposition yields three eigenvalues ​​λ1, λ2, and λ3 and their corresponding eigenvectors ν1, ν2, and ν3, where λ1 ≤ λ2 ≤ λ3.

[0031] Step (3): The eigenvector ν1 corresponding to the smallest eigenvalue λ1 is the normal vector direction. By normalizing ν1, the point can be obtained. normal vector :

[0032] That is, the eigenvector corresponding to the smallest eigenvalue is a point. The normal vectors are obtained, thus yielding a large number of normal vectors.

[0033] Step (4), place the point Normalization of the normal vector.

[0034] Point Connect each of its k nearest neighbors with a line, and refer to... Figure 3 (When k is 5), a k-nearest neighbor graph is formed. All k-nearest neighbor graphs constitute the global k-nearest neighbor graph.

[0035] The Prim algorithm is used to process the global k-nearest neighbor graph to generate the minimum spanning tree.

[0036] At all points If a root is randomly selected and its normal vector does not point towards the origin of the camera coordinate system, then the normal vector of the root is reversed.

[0037] Traverse the minimum spanning tree, starting from the root (which is the parent node). Compare the normal vector of each child node with the normal vector of the parent node. If the angle between the two normal vectors is greater than 90°, reverse the normal vector of the child node. This results in a globally consistent normal vector field, meaning that the normal vector of each point in the point cloud points to the same side (either outward or in the direction of the camera).

[0038] The fourth step is to calculate the divergence of the normal vector field of the point cloud and extract the weld point cloud based on the divergence.

[0039] For each point, the K-nearest neighbor algorithm searches for m nearest neighbor points within its neighborhood. The number of points in the resulting neighborhood set is smaller than the number of points in the neighborhood set in step (1) above.

[0040] The divergence at each point is calculated using the following formula. :

[0041] In the formula, for The nearest neighbor, For point The positional difference between it and its nearest neighbor indicates the distance from Point of view Position vector. For point The distance between it and its nearest neighbors. For point The vector difference between the normal vector of a point and the normal vector of its nearest neighbor. This is the projection of the position vector onto the direction of the change of the normal vector.

[0042] For each point in the point cloud of the weldment, the divergence value corresponding to that point is calculated based on the normal vector distribution of its neighboring points, in order to characterize the local geometric properties of the normal vector field at that point.

[0043] Specifically, such as Figure 4As shown in Figure (a), when the normal vector in the neighborhood of a point shows an outward divergence trend, the divergence calculation result at that point is positive. The absolute value is used to characterize the degree of divergence of the normal vector. The larger the absolute value, the more drastic the surface geometry change and the more obvious the divergence characteristics. Figure 4 As shown in Figure (b), when the normal vectors in the neighborhood of a point show an inward convergence trend, the divergence calculation result at that point is negative. The magnitude of its absolute value is used to characterize the degree of convergence of the normal vectors. The larger the absolute value, the more obvious the surface depression or broken line structure at that point.

[0044] Since the weld area is usually located at the junction of two or more weldment surfaces, its local geometry exhibits obvious normal vector convergence characteristics. Therefore, the divergence value of the corresponding point is usually negative, and its absolute value is significantly greater than that of the non-weld area.

[0045] Based on the above characteristics, this embodiment adopts a weld region determination method based on divergence threshold: a divergence threshold A is preset, where A is a negative number; for any point in the weldment point cloud, when the divergence value of that point... If the value is less than the threshold A, the point is identified as a weld point; otherwise, it is identified as a non-weld point. By applying the above judgment rule to all points, the weld area and the non-weld area are effectively separated, thereby obtaining the weld point cloud.

[0046] For example, the threshold A can be set to -0.2, but it is not limited to this value. Its specific value can be set according to the point cloud density, noise level and the geometric scale of the weldment.

[0047] Figure 5 It is a segmented linear weld point cloud. Figure 6 It is a cloud of points representing the segmented arc-shaped weld seam.

[0048] It is evident that the weld seam point cloud can be accurately extracted, and the extracted weld seam point cloud is relatively uniformly distributed and has a continuous shape.

[0049] After completing the weld point cloud extraction process described above, the weld trajectory is generated based on the weld point cloud. Based on the actual structure of the workpiece, the obtained set of weld point clouds may include multiple weld segments. To better fit the weld, these weld segments need to be divided into separate point cloud clusters. These point cloud clusters need to meet two conditions: first, they correspond to only one typical geometric configuration (straight line, circular arc, or free-form smooth curve); second, each point cloud cluster contains only one weld segment of that typical geometric configuration. To achieve accurate separation of weld segments, the following clustering method is proposed.

[0050] The first step is to project the point cloud of the weldment and the extracted weld seam point cloud onto multiple viewpoints to obtain two-dimensional projection images corresponding to each viewpoint.

[0051] Step (1): Calculation of principal axis direction and construction of coordinate system.

[0052] To ensure that the generated projection viewpoint accurately matches the actual geometry of the weldment, the principal axis direction calculation is first performed on the point cloud of the weldment. The specific process is as follows: The Random Sample Consensus (RANSAC) algorithm was used to perform planar fitting on the point cloud of the weldment, resulting in a planar model composed of a set of interior points. This planar model was defined as the "principal plane." Planar normal vectors were extracted based on this principal plane and set as the third principal axis direction of the weldment's oriented bounding box, i.e., the Z-axis direction. To ensure the consistency of the coordinate system orientation, when the Z-axis component of the plane normal vector was negative, its direction was reversed.

[0053] Next, a two-dimensional coordinate system is constructed within the main plane. The world coordinate system of the weldment point cloud itself is as follows: Figure 7 As shown, from the two reference direction vectors of the world coordinate system, the X-axis direction vector and the Y-axis direction vector, one that is not parallel to the plane normal vector is selected as a preset reference direction vector. This preset reference direction vector is projected onto the principal plane to form a projection vector in the principal plane. This projection vector is used as the X-axis direction vector with the principal plane as the reference. Then, the Y-axis direction vector is obtained by performing a cross product operation between the plane normal vector and the X-axis direction vector. This constructs a three-dimensional orthogonal coordinate system with the principal plane as the reference, which satisfies the right-hand coordinate system constraints.

[0054] To improve the stability and reliability of the principal axis direction, the interior points of the principal plane are projected onto the XY plane in the constructed 3D orthogonal coordinate system based on the principal plane, forming projection points. Then, principal component analysis (PCA) is performed on all projection points in this XY plane. The direction corresponding to the first principal component is used as the updated X-axis direction, and the Y-axis direction is corrected based on the cross product of the updated X-axis and Z-axis directions. This results in a new principal axis coordinate system, as follows: Figure 8 As shown.

[0055] Step (3): After obtaining the new principal axis coordinate system, calculate the transformation matrix from the world coordinate system to the new principal axis coordinate system, and transform the weldment point cloud to the new principal axis coordinate system using the transformation matrix.

[0056] Step (4): Calculate the axis-aligned bounding box of the weldment point cloud in the new principal axis coordinate system using the projection extremum method. The process of generating the axis-aligned bounding box mainly involves projecting each 3D point in the weldment point cloud onto the three principal axis directions, finding the minimum and maximum values ​​in the three projection value sets, and directly constructing the axis-aligned bounding box based on the six found extrema. The axis-aligned bounding box is shown below. Figure 9 As shown.

[0057] Step (5): Constructing a set of multi-view projection perspectives.

[0058] Set the center point of the axis-aligned bounding box as the projection reference center, and set projection viewpoints along each axis of the new principal axis coordinate system to form multiple orthogonal projection viewpoints. Specifically, set three orthogonal projection viewpoints, such as... Figure 10 As shown, the first principal axis positive direction is formed along the positive X-axis, the second principal axis positive direction is formed along the positive Y-axis, and the third principal axis positive direction is formed along the positive Z-axis.

[0059] For each projection viewpoint, a camera coordinate system is created with the projection reference center as the camera target point and the direction of that projection viewpoint as the optical axis direction. The distance between the camera position and the projection reference center is several times the diagonal length of the axis-aligned bounding box; for example, the distance between the camera and the projection reference center (i.e., the camera target point) is twice the diagonal length of the axis-aligned bounding box. Setting the distance between the camera and the projection reference center ensures that the weldment point cloud is completely rendered in the corresponding projection plane, without any missing areas. The camera here is not a real camera.

[0060] This results in three camera coordinate systems.

[0061] Step (6): Establish multiple projection field ranges based on each camera coordinate system.

[0062] The projected field of view is established for each of the three camera coordinate systems. The width of the projected field of view is... W The height is H ,width W It is the maximum value of the X-axis coordinate. Subtract the minimum value of the X-axis coordinate ,Right now .high H It is the maximum value of the Y-axis coordinate. Subtract the minimum value of the Y-axis coordinate ,Right now: The projection field of view is a grid plane, and the size of each grid can be 0.5×0.5mm.

[0063] This results in three projection fields of view.

[0064] Step (7): Calculate the translation vector T and rotation matrix R of the rigid body transformation from the new principal axis coordinate system to each camera coordinate system.

[0065] Step (8): Generate a two-dimensional projection image.

[0066] First, the weldment point cloud is transformed from the new principal axis coordinate system to the camera coordinate system using the following formula: In the formula, For the points in the point cloud of the weldment, It is a point transformed to the camera coordinate system. The coordinates in the camera coordinate system are ( Xc , Yc , Zc ).

[0067] Next, orthographic projection and pixel mapping are performed. The points... Orthographic projection is applied to the projection field of view, forming pixels within that field of view, which then fall into a grid. It is evident that orthographic projection directly discards the depth component. Z C .

[0068] The coordinates of the obtained pixel are ( u , v ):

[0069] It should be noted that regarding the points from the point cloud of the weldment... The transformation process down to the pixel, the total transformation matrix yes:

[0070] In the formula, The transformation matrix is ​​3×3. The resulting pixels are... .

[0071] When multiple pixels exist at the same grid location, the corresponding depth component is retained. Z C Delete the smallest pixel.

[0072] Then, the weld point cloud is orthographically projected onto the projection field of view using the same method described above, forming pixels within the projection field of view. This yields a two-dimensional projected image. Because there are three projection fields of view—left view (positive x-axis direction), front view (positive y-axis direction), and top view (positive z-axis direction)—there are three two-dimensional projected images. k is used as a number, and the value of k is 1, 2, or 3. One projection field of view corresponds to one two-dimensional projection image.

[0073] Step (9): Color and draw the two-dimensional projection image.

[0074] In the generated 2D projection image, a first color is used to color the pixels projected from the weldment point cloud, and a second color, which is significantly different from the first color, is used to color the pixels projected from the weld seam point cloud. This highlights the weld seam area in the 2D projection image, resulting in a 2D projection image after color rendering. This facilitates subsequent weld seam identification and processing. There are three 2D projection images after color rendering. .

[0075] refer to Figure 12 Displaying two-dimensional projection images of a workpiece after color rendering from three perspectives. . Figure 12 VC1 is left-facing, VC2 is top-facing, and VC3 is forward-facing.

[0076] refer to Figure 11 The image shows a two-dimensional projection image of another workpiece after color rendering processing from three perspectives.

[0077] Multiple 2D projection images generated from different viewpoints after color rendering processing This provides a stable and consistent geometric basis for subsequent weld seam recognition and multi-view fusion, improving the reliability and applicability of the overall technical solution.

[0078] The second step is to process the colored two-dimensional projection image. Input the YOLOv8-Seg weld seam recognition instance segmentation model, and generate a 2D projection image of the YOLOv8-Seg weld seam recognition instance segmentation model after color rendering. Instance segmentation is performed, dividing each weld seam segment in the image into closed polygons (i.e., each weld seam segment is surrounded by a polygonal instance segmentation box, the shape of which matches the shape of the weld seam segment; if the weld seam segment is straight, the polygonal instance segmentation box is also straight, and if the weld seam segment is curved, the polygonal instance segmentation box is also curved). The weld seams are then classified according to their shape (specifically, straight or curved), resulting in the instance segmentation image. Each instance segmentation result image Label containing multiple polygon instance separators k i Label k i ={Label k 1 Label k 2 Label k 3 ,…,Label k n}, where n is the number of polygon instance bounding boxes. Label k i It includes the weld segment category information TYPE corresponding to the i-th polygon instance segmentation box, and the vertices of the polygon instance segmentation box in the two-dimensional image. The position coordinates in the middle.

[0079] There are 3 two-dimensional projected images There are 3 instance segmentation result images. .like Figure 14 and Figure 15 As shown, because there are 3 ( , , This generates three instance segmentation result images. , , .

[0080] Two-dimensional projection image after coloring and drawing After being input into the YOLOv8-Seg weld seam recognition and segmentation model, the model first extracts features from the image through a backbone network, generating multi-scale feature maps. Then, a feature fusion structure integrates features from different scales to enhance the ability to represent the slender structure and local details of the weld seam. Based on this, the network simultaneously performs target category prediction, bounding box regression, and pixel-level segmentation mask generation on the feature maps, achieving end-to-end recognition and segmentation of the weld seam target. Model output results. It includes the location information, category information and corresponding segmentation mask of the weld in the two-dimensional image, which are used to characterize the spatial distribution range of the weld in the image plane.

[0081] The YOLOv8-Seg weld recognition instance segmentation model can achieve accurate segmentation of the weld region in the 2D projection image after color rendering, providing reliable 2D constraint information for subsequent weld point cloud projection, fusion, and 3D weld modeling.

[0082] The process of constructing the YOLOv8-Seg weld seam recognition instance segmentation model is as follows: Prepare a large number of 2D projected images with color rendering from different viewpoints, and manually annotate the weld seam regions in the images. The annotations include weld seam category information and its corresponding pixel-level segmentation region. The weld seam category is used to distinguish different geometric shapes of weld seams in the 2D image, including but not limited to straight and curved weld seams. Based on the annotated data, perform data augmentation processing on the images to improve the model's adaptability to changes in pose, scale, and imaging conditions.

[0083] A pre-trained network model was used as the base model, and the initial training was completed using the COCO8 dataset publicly available on the Ultralytics website, so that the network has general object representation and segmentation capabilities.

[0084] Based on the initial training, the network is trained by transfer learning using manually labeled two-dimensional projection images, enabling the model to adaptively adjust features for weld targets, thus maintaining good segmentation and recognition performance even with a limited number of samples.

[0085] The third step involves using the instance segmentation results output by the YOLOv8-Seg weld recognition instance segmentation model to divide the weld point cloud containing multiple weld segments into multiple point cloud clusters containing only one weld segment. This provides an accurate point set foundation for subsequent differential geometric fitting.

[0086] Step S301: The two-dimensional projected images obtained from each viewpoint have been projected in the previous steps. Perform instance segmentation on multiple instance segmentation result images. Polygon instance divider label k i The instances are permuted and combined to generate instance combinations (P1, P2, P3). P1 is a set of polygon instance segmentation boxes selected from the first instance segmentation result image VCF1. Figure 14 and 15 For example, the set of values ​​for P1 is {Label1}. 1}. P2 is the set of polygonal instance segmentation boxes selected from the VCF2 image of the second instance segmentation result. The value set of P2 is {Label2}. 1 Label2 2 Label2 3 ...Label2 8}. P3 is the set of polygonal instance segmentation boxes selected from the VCF3 image of the third instance segmentation result. The value set of P3 is {Label3}. 1 For example, when P1 takes (Label1) from VCF1... 1 P2 takes (Label2) from VCF2. 1 P3 takes (Label3) from VCF3. 1 ), then generate an instance composition (Label1) 1 Label2 1 Label3 1 P1 takes (Label1) from VCF1. 1 P2 takes (Label2) from VCF2. 2 P3 takes (Label3) from VCF3. 1 ), then generate an instance composition (Label1) 1 Label2 2 Label3 1 As can be seen, each instance combination is generated from the Label in each image.k i Select one, and so on, to generate a total of 8 instance combinations.

[0087] Step S302: Perform clustering processing on the weld seam point cloud to obtain the clustering results.

[0088] Iterate through all instance combinations. For each instance combination, set the three... Find the intersection of the polygon instance segmentation bounding box and the weld point cloud, and simultaneously find the points that fall within each polygon instance segmentation bounding box. Medium polygonal frame instance box Label k i The points in the diagram can be grouped into a single weld cluster.

[0089] Based on the instance combination (P1, P2, P3), the type of the categorized independent weld cluster can be determined. The category of the categorized independent weld cluster is determined by the TYPE information in P1, P2, and P3. If the TYPE information in any one of P1, P2, and P3 is a curve, it is considered a curve cluster. If all the TYPE information in P1, P2, and P3 is a straight line, it is considered a straight line cluster.

[0090] For example, in the case of 8 instance combinations, instance combination (Label1) 1 Label2 1 Label3 1 These correspond to the polygonal box instance box 1 (Label1) in VCF1. 1 ), the polygon instance box 1 (Label2) in VCF2 1 ), Polygon instance box 1 (Label3) in VCF3 1 Based on the spatial mapping parameters saved in the first step and step (8) stage of "orthophoto projection of weld point cloud to projection field of view", the points Cloud in the weld point cloud are projected onto VCF1, VCF2, and VCF3 respectively, and the points Cloud that can simultaneously satisfy the following three conditions are found: Condition ①: The polygon box instance 1 (Label1) falls within VCF1. 1 ); Condition ②: The polygon instance box 1 (Label2) falls within VCF2. 1 ); Condition ③: The polygon instance box 1 (Label3) falls within VCF3. 1 ).

[0091] Points Cloud that meet the above three conditions are grouped into an independent weld cluster. The category of this cluster depends on the category of the corresponding instance. Because in instance combination (Label1) 1 Label21 Label3 1 In the context of the combination (P1, P2, P3), at least one of P contains a curve based on the logic that Label2's TYPE information is a curve. 1 The TYPE information is a curve, therefore the category of this cluster is a curve.

[0092] For another example, instance composition (Label1) 1 Label2 3 Label3 1 These correspond to the polygonal box instance box 1 (Label1) in VCF1. 1 ), VCF2 polygonal box instance box 3 (Label2) 3 ), VCF3 polygonal box instance box 1 (Label3) 1 Project the points Cloud in the weld seam point cloud onto VCF1, VCF2, and VCF3 respectively, and find the points Cloud that simultaneously satisfy the following three conditions: Condition ①: The polygon box instance 1 (Label1) falls within VCF1. 1 ); Condition ②: The polygon instance box 3 (Label2) falls within VCF2. 3 ); Condition ③: The polygon instance box 1 (Label3) falls within VCF3. 1 ).

[0093] Points Cloud that meet the above three conditions are grouped into an independent weld cluster. The category of this cluster depends on the category of the corresponding instance. Because in instance combination (Label1) 1 Label2 3 Label3 1 In the case of P, according to the judgment logic that the TYPE information in all P of the combination (P1, P2, P3) is a straight line, the category of this cluster is a straight line.

[0094] Similarly, in the above example, the points on the weld can ultimately generate eight independent weld clusters. These eight independent weld clusters exhibit strong integrity and continuity.

[0095] Each final 3D weld cluster corresponds to a set of weld point indexes and determined weld category information, which can be linear or curved.

[0096] Subsequently, points in each cluster can be fitted with straight lines or curves according to their type, and a continuous weld space trajectory can be generated based on the weld sub-cluster point cloud.

[0097] The fourth step is to generate the spatial trajectory of the weld seam based on the weld seam cluster.

[0098] Each weld point cluster is fitted to obtain a fitted line.

[0099] Then, the fitted line is sampled at equal intervals to obtain a spatial coordinate sequence composed of discrete points, thus obtaining the weld trajectory. The spatial coordinate points are located in the coordinate system of the robotic arm's end effector, which can accurately reflect the shape of the weld segment in three-dimensional space.

[0100] After obtaining the weld seam trajectory, the spatial coordinates are input into the robot arm inverse kinematics solution module to calculate the rotation angle of the robot arm joints corresponding to each spatial coordinate point, thus forming an executable joint motion sequence for the robot arm. Combined with conventional robot arm control methods, this enables the robot arm to perform continuous, smooth, and high-precision movements along the weld seam trajectory, thereby achieving automated welding operations. It is evident that this not only ensures the accuracy and continuity of the welding path but also provides reliable data support for welding path planning, robot operation control, and weld quality inspection. The fitted weld seam spatial trajectory closely matches the actual weld seam morphology, meeting the trajectory guidance requirements for automated, high-precision welding.

[0101] Step (1): Use the RANSAC algorithm to fit the straight line model and the circular arc model for each weld point cluster. Under the condition of fitting the straight line model and the circular arc model, select the model with the most internal points as the correct model. The criterion for the most internal points is that the proportion of internal points reaches or exceeds the preset threshold (for example, the threshold value is 90%).

[0102] Save the fitting parameters of the correct model. The line type corresponding to the correct model is the fitted line of the weld segment.

[0103] For weld subclusters that do not meet the fitting conditions of linear model and circular arc model, they are identified as irregular curved welds. The B-spline curve fitting method is used to smooth them. By optimizing the control points, it is ensured that the fitted curve can pass through or approach all points in the cluster, so as to obtain a continuous and smooth B-spline curve.

[0104] Step (2) involves sampling each fitted line obtained in step (1) at equal intervals to obtain a spatial coordinate sequence composed of discrete points, thus obtaining the weld trajectory. The spatial coordinate points are located in the coordinate system of the robotic arm's end effector, which can accurately reflect the shape of the weld segment in three-dimensional space. Figure 16 , 17 The resulting weld seam trajectory diagram.

[0105] It should be noted that if a set of weld point clouds contains only one weld segment, then the aforementioned first step of using a clustering method based on feature similarity to separate a set of weld point clouds is unnecessary. Instead, curve fitting can be directly performed on the weld segment to obtain the fitted line.

[0106] After obtaining the weld seam trajectory, the spatial coordinates are input into the robot arm inverse kinematics solution module to calculate the rotation angle of the robot arm joints corresponding to each spatial coordinate point, thus forming an executable joint motion sequence for the robot arm. Combined with conventional robot arm control methods, this enables the robot arm to perform continuous, smooth, and high-precision movements along the weld seam trajectory, thereby achieving automated welding operations. It is evident that this not only ensures the accuracy and continuity of the welding path but also provides reliable data support for welding path planning, robot operation control, and weld quality inspection.

[0107] It is evident that the above method is applicable to complex weld structures and can fully leverage the operational precision and efficiency advantages of automated welding equipment.

[0108] It should be noted that for weld point clouds containing only one weld segment (e.g., only one straight weld segment or only one curved weld segment), the instance segmentation results output by the YOLOv8-Seg weld recognition instance segmentation model in step three are used to divide the weld point cloud containing only one weld segment into point cloud clusters, which is achieved in the following way: In instance segmentation results When there is only one weld seam area in the image, the instance segmentation result image The image contains a polygonal instance segmentation box, which includes weld segment category information. For weld point clouds with only one weld segment, clustering is performed, and the points in the weld point cloud are respectively mapped to the instance segmentation result image. By projecting upwards, the points within the polygonal bounding boxes that simultaneously fall into the segmentation result image of each instance are grouped into point cloud clusters, resulting in the final weld point clusters.

Claims

1. A method for clustering and optimizing weld seam point clouds, characterized in that, Includes the following steps: Step (1): Establish multiple projection field ranges based on orthogonal projection perspectives; Step (2): The weld point cloud and the weld point cloud including multiple weld seam segments are orthophoto-projected onto the projection field of view, and pixels are formed in the projection field of view to obtain a two-dimensional projection image; since there are multiple projection field of view, multiple two-dimensional projection images are obtained. Step (3): Color the multiple two-dimensional projection images. Use a first color to color the pixels obtained from the weldment point cloud projection, and use a second color that is significantly different from the first color to color the pixels obtained from the weld seam point cloud projection, to obtain the two-dimensional projection image after color rendering. ; Step (4) Render the two-dimensional projection image after coloring and drawing. Input the YOLOv8-Seg weld seam recognition instance segmentation model, and generate a 2D projection image of the YOLOv8-Seg weld seam recognition instance segmentation model after color rendering. Perform instance segmentation and output the instance segmentation result image. The instance segmentation result image Label containing multiple polygon instance separators k i Label k i ={Label k 1 Label k 2 Label k 3 ,…,Label k n }, where n represents the number of polygon instance bounding boxes, and Label k i Includes the weld segment category information TYPE corresponding to the i-th polygon instance segmentation box; Step (5): Based on the instance segmentation result image The weld point cloud containing multiple weld segments is divided into multiple point cloud clusters containing only one weld segment; Step (5-1) segment multiple instance result images Polygon instance divider label k i Perform permutations and combinations to generate instance combinations (P1, P2, ..., PK), where the set of values ​​for P1 is {Label1}. i }, the set of values ​​for P2 is {Label2} i }, the set of values ​​for PK is {Label k i }; Step (5-2) involves clustering the weld point cloud containing multiple weld segments, and then segmenting the points in the weld point cloud into the instance segmentation result image. Up projection, find the segmentation result image that falls into each instance simultaneously. Medium polygonal frame instance box Label k i The points in the cloud are grouped into point cloud clusters. All instance combinations (P1, P2, ..., PK) are traversed. If the TYPE information in any item of (P1, P2, ..., PK) is the first type of weld segment information, then the type of the point cloud cluster is determined to be the first type of weld segment information cluster. If the TYPE information in each item of (P1, P2, ..., PK) is the second type of weld segment information, then the type of the point cloud cluster is determined to be the second type of weld segment information cluster.

2. The clustering optimization method for weld point clouds according to claim 1, characterized in that, The first type of weld segment information is a curve, and the second type of weld segment information is a straight line.

3. The clustering optimization method for weld point clouds according to claim 1, characterized in that, The process of establishing multiple projection field ranges based on orthogonal projection perspectives in step (1) is carried out in the following manner: Step 1): Calculation of principal axis direction and construction of coordinate system; The point cloud of the weldment is fitted with a random sampling consensus algorithm to obtain a principal plane. The plane normal vector is extracted based on the principal plane and set as the Z-axis direction of the weldment's oriented bounding box. From the X-axis direction vector and Y-axis direction vector of the world coordinate system where the weldment point cloud is located, select one that is not parallel to the plane normal vector as a preset reference direction vector; project this preset reference direction vector onto the main plane to form a projection vector in the main plane, and use this projection vector as the X-axis direction vector with the main plane as the reference. Then, obtain the Y-axis direction vector by performing a cross product operation between the plane normal vector and the X-axis direction vector, and construct a three-dimensional orthogonal coordinate system with the main plane as the reference. Projecting the interior points of the principal plane onto the XY plane in the three-dimensional orthogonal coordinate system based on the principal plane forms projection points; Then, principal component analysis is performed on all projection points in the XY plane. The direction corresponding to the first principal component is taken as the updated X-axis direction. Based on the cross product relationship between the updated X-axis direction and the Z-axis direction, the Y-axis direction is corrected to obtain the new principal axis coordinate system. Step 2), calculate the transformation matrix from the world coordinate system to the new principal axis coordinate system, and transform the point cloud of the weldment to the new principal axis coordinate system using the transformation matrix; Step 3), calculate the axis-aligned bounding box of the weldment point cloud in the new principal axis coordinate system using the projection extremum method; Step 4) Set the center point of the axis-aligned bounding box as the projection reference center, and set the projection viewpoints along each axis of the new principal axis coordinate system to form multiple orthogonal projection viewpoints; For each projection viewpoint, a camera coordinate system is created with the projection reference center as the camera target point and the direction of that projection viewpoint as the optical axis direction. The distance between the camera position and the projection reference center is several times the length of the diagonal of the axis-aligned bounding box, and multiple camera coordinate systems are created. Step 5) Establish multiple projection field ranges based on each camera coordinate system.

4. The clustering optimization processing method for weld point clouds according to claim 3, characterized in that, In step (2), the following formula is used: In the formula, T is the translation vector and R is the rotation matrix; The weldment point cloud and the weld point cloud including multiple weld segments are transformed from the new principal axis coordinate system to the camera coordinate system. Then, the weldment point cloud and the weld point cloud including multiple weld segments in the camera coordinate system are orthophotographed onto the projection field of view.

5. The clustering optimization method for weld point clouds according to claim 3, characterized in that, There are three orthogonal projection perspectives: the first principal axis positive direction is formed along the positive X-axis, the second principal axis positive direction is formed along the positive Y-axis, and the third principal axis positive direction is formed along the positive Z-axis.

6. A method for clustering and optimizing weld seam point clouds, characterized in that, Includes the following steps: Step (1): Establish multiple projection field ranges based on orthogonal projection perspectives; Step (2): The point cloud of the weldment and the point cloud of the weld with only one weld segment are orthophotos projected onto the projection field of view, and pixels are formed in the projection field of view to obtain a two-dimensional projection image; since there are multiple projection field of view, multiple two-dimensional projection images are obtained. Step (3): Color the multiple two-dimensional projection images. Use a first color to color the pixels obtained from the weldment point cloud projection, and use a second color that is significantly different from the first color to color the pixels obtained from the weld seam point cloud projection, to obtain the two-dimensional projection image after color rendering. ; Step (4) Render the two-dimensional projection image after coloring and drawing. Input the YOLOv8-Seg weld seam recognition instance segmentation model, and generate a 2D projection image of the YOLOv8-Seg weld seam recognition instance segmentation model after color rendering. Perform instance segmentation and output the instance segmentation result image. The instance segmentation result image It contains a polygon instance segmentation box, and the polygon instance segmentation box contains weld segment category information; Step (5) involves clustering the weld point cloud with only one weld segment, and then segmenting the points in the weld point cloud into the instance segmentation result image. Project upwards, find the points that fall within the polygonal bounding box of each instance segmentation result image, and group these points into point cloud clusters.

7. The clustering optimization processing method for weld point clouds according to claim 6, characterized in that, The process of establishing multiple projection field ranges based on orthogonal projection perspectives in step (1) is carried out in the following manner: Step 1): Calculation of principal axis direction and construction of coordinate system; The point cloud of the weldment is fitted with a random sampling consensus algorithm to obtain a principal plane. The plane normal vector is extracted based on the principal plane and set as the Z-axis direction of the weldment's oriented bounding box. From the X-axis direction vector and Y-axis direction vector of the world coordinate system where the weldment point cloud is located, select one that is not parallel to the plane normal vector as a preset reference direction vector; project this preset reference direction vector onto the main plane to form a projection vector in the main plane, and use this projection vector as the X-axis direction vector with the main plane as the reference. Then, obtain the Y-axis direction vector by performing a cross product operation between the plane normal vector and the X-axis direction vector, and construct a three-dimensional orthogonal coordinate system with the main plane as the reference. Projecting the interior points of the principal plane onto the XY plane in the three-dimensional orthogonal coordinate system based on the principal plane forms projection points; Then, principal component analysis is performed on all projection points in the XY plane. The direction corresponding to the first principal component is taken as the updated X-axis direction. Based on the cross product relationship between the updated X-axis direction and the Z-axis direction, the Y-axis direction is corrected to obtain the new principal axis coordinate system. Step 2), calculate the transformation matrix from the world coordinate system to the new principal axis coordinate system, and transform the point cloud of the weldment to the new principal axis coordinate system using the transformation matrix; Step 3), calculate the axis-aligned bounding box of the weldment point cloud in the new principal axis coordinate system using the projection extremum method; Step 4) Set the center point of the axis-aligned bounding box as the projection reference center, and set the projection viewpoints along each axis of the new principal axis coordinate system to form multiple orthogonal projection viewpoints; For each projection viewpoint, a camera coordinate system is created with the projection reference center as the camera target point and the direction of that projection viewpoint as the optical axis direction. The distance between the camera position and the projection reference center is several times the length of the diagonal of the axis-aligned bounding box, and multiple camera coordinate systems are created. Step 5) Establish multiple projection field ranges based on each camera coordinate system.

8. The clustering optimization processing method for weld point clouds according to claim 7, characterized in that, In step (2), the following formula is used: In the formula, T is the translation vector and R is the rotation matrix; Transform the weldment point cloud and weld seam point cloud from the new principal axis coordinate system to the camera coordinate system, and then orthophoto project the weldment point cloud and weld seam point cloud in the camera coordinate system onto the projection field of view.

9. A method for generating weld seam trajectories, characterized in that, The point cloud cluster according to any one of claims 1-8 generates the spatial trajectory of the weld seam.

Images