Interactive weld joint identification method and system based on three-dimensional point cloud

Abstract

The invention relates to an interactive welding seam identification method and system based on a three-dimensional point cloud, and belongs to the technical field of welding seam identification, and the method comprises the following steps: 1, executing human-computer interaction; wherein data acquisition and prior setting are carried out; 2, executing a self-adaptive recognition algorithm; wherein the identification module is dynamically selected or combined according to the obtained prior type, the point cloud quality index and the annotation information; step 3, outputting an identification result and generating the pose of the mechanical arm; the method has the beneficial effect that the welding seam recognition precision is improved by adopting a mode of priori-driven man-machine interaction, self-adaptive multi-algorithm recognition and robot output.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of weld seam recognition of welding, and particularly relates to an interactive weld seam recognition method and system based on three-dimensional point clouds. BACKGROUND

[0002] Weld seam recognition technology is one of the core links for automatic welding robots to realize autonomous operation, and its accuracy and stability directly determine the reliability of welding path planning and execution, and greatly affect the performance level and market competitiveness of welding robots. Only with high-precision and high-robustness weld seam recognition capability, welding robots can adapt to complex workpiece structures and industrial environments, so as to truly realize intelligent and flexible automatic welding operation.

[0003] Existing weld seam recognition methods mainly rely on two-dimensional image processing or three-dimensional point cloud geometric feature extraction based on fixed rules. The two-dimensional image method is limited by light, reflection and complex texture of the weld seam surface, and has poor recognition accuracy and robustness, and is difficult to adapt to diversified weld seam shapes in industrial environments. Although the traditional three-dimensional point cloud method can obtain the spatial geometric information of the weld seam, it generally has the following problems: first, the algorithm interaction is insufficient, and fixed threshold, rule fitting and other methods are usually used, and there is lack of human-computer collaborative adjustment means, resulting in poor recognition effect when complex workpiece structures or weld seams are blocked or noisy points; second, the point cloud data is large, and the existing processing flow is inefficient, which is difficult to meet the real-time or rapid iteration requirements; third, the recognition result lacks intuitive visualization and interactive operation, and users cannot flexibly correct and confirm based on the recognition result, thereby affecting the accuracy and reliability of subsequent welding path planning and robot operation. SUMMARY

[0004] The application provides an interactive weld seam recognition method and system based on three-dimensional point clouds, which is used to solve the technical problem of improving the recognition accuracy of the weld seam, and adopts the mode of prior-driven human-computer interaction + adaptive multi-algorithm recognition + robot output to improve the recognition accuracy of the weld seam.

[0005] In order to achieve the above purpose, the application realizes the following technical scheme:

[0006] An interactive weld seam recognition method based on three-dimensional point clouds, comprising the following steps:

[0007] Step 1: performing human-computer interaction; wherein data acquisition and prior setting are performed;

[0008] Step 2: performing adaptive recognition algorithm; wherein according to the obtained prior type, point cloud quality index and labeling information, the recognition module is dynamically selected or combined;

[0009] Step 3: recognition result output and robot pose generation.

[0010] Optionally, in step one, the steps of data acquisition and preprocessing, human-computer interaction interface and ROI determination, prior information initialization and quality assessment, and interactive parameter adjustment and rapid preview are performed.

[0011] Optionally, in the data acquisition and preprocessing step, a structured light camera is used to scan the workpiece to obtain a high-density three-dimensional point cloud, which contains x, y and z coordinates and intensity / confidence information.

[0012] The preliminary processing of the original point cloud includes outlier removal, voxel grid downsampling, normal vector estimation, and curvature / convexity calculation. The normal vector estimation is based on neighborhood PCA, with neighborhood radius r or neighborhood point k configuration.

[0013] Optionally, in the human-computer interaction interface and ROI determination step, based on point cloud visualization, a variety of interactive selection tools are provided on the three-dimensional view to label or correct the target recognition area.

[0014] Optionally, in step two, the double-plane intersection fitting module is used for cases with obvious groove / face, the RANSAC algorithm is used to fit the normal vectors of the two sides, the intersection direction is calculated, and then the least squares method is used to accurately solve the centerline position in the intersection neighborhood.

[0015] Optionally, in step two, the cross-sectional profile analysis module is used for V-shaped, U-shaped groove or obvious symmetrical face welds, equal-interval slices are made along the candidate centerline direction, cross-sectional contour fitting is performed, and the width, depth and cross-sectional asymmetry are estimated.

[0016] Optionally, in step two, the curvature / spine line extraction module is used for surface curvature prominent structures such as fillet welding, surfacing and fillet, and the spine line or groove centerline is extracted through point cloud normal and second-order curvature information.

[0017] Optionally, in step two, the template matching / registration module is used for known standardized weld template and repetitive workpiece, local descriptor matching is used and ICP is used for accurate registration to obtain accurate centerline.

[0018] Optionally, in step three, the steps include basic geometric output, welding gun pose generation, output data format and interface, and risk prompt and suggested action.

[0019] An interactive weld recognition system based on three-dimensional point cloud, comprising:

[0020] A human-computer interaction module for performing human-computer interaction;

[0021] A data processing module connected to the human-computer interaction module for performing adaptive recognition algorithm;

[0022] The output module, connected to the data processing module, is used for outputting recognition results and generating the robot arm pose.

[0023] The beneficial effects of this invention are:

[0024] This invention proposes a holistic solution of "prior-driven human-computer interaction + adaptive multi-algorithm recognition + robot-friendly output." Prior knowledge is introduced into the recognition process by manually confirming the workpiece type and ROI; recognition algorithms are dynamically selected / fused based on prior knowledge and point cloud quality indicators; and the recognition results are directly mapped to robot welding torch pose and process suggestions considering accessibility and collision. Compared to existing technologies, this invention improves recognition accuracy and robustness (with better adaptability to noisy, occluded, and multi-weld workpieces); shortens manual adjustment and on-site deployment time (interactive initialization + local recalculation reduces repetitive global calculations); and directly provides executable pose and parameter suggestions for the welding robot, reducing integration and debugging costs, improving welding quality consistency, and thus enhancing the market competitiveness of automated welding robots. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a system structure diagram of the present invention;

[0027] Figure 2 This is a flowchart of the process of the present invention. Detailed Implementation

[0028] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0029] Example 1;

[0030] like Figure 1 As shown, this embodiment provides an interactive weld seam recognition system based on three-dimensional point clouds, including:

[0031] The human-computer interaction module is used to perform human-computer interaction.

[0032] The data processing module, connected to the human-computer interaction module, is used to execute adaptive recognition algorithms;

[0033] The output module, connected to the data processing module, is used for outputting recognition results and generating the robot arm pose.

[0034] Example 2;

[0035] Based on Example 1, such as Figure 2 As shown, this embodiment provides an interactive weld seam recognition method based on three-dimensional point clouds, including the following steps:

[0036] Step 1: Perform human-computer interaction; this includes data collection and prior settings.

[0037] 1. Data acquisition and preprocessing;

[0038] An active structured light camera is used to scan the workpiece to obtain a high-density 3D point cloud (the point cloud contains x, y, z coordinates and intensity / confidence information).

[0039] Preliminary processing of the original point cloud includes: statistical outlier removal, voxel grid downsampling, normal vector estimation (based on neighborhood PCA, with configurable neighborhood radius r or number of neighboring points k), and curvature / convexity calculation.

[0040] 2. Determining the human-computer interaction interface and ROI;

[0041] Based on point cloud visualization in a 3D view, it provides a variety of interactive selection tools (box selection, brush, polygon selection based on planar projection, and single-point expansion) for operators to annotate or correct the target recognition region (Region of Interest, ROI).

[0042] The system provides a drop-down or icon selection for the workpiece "type / model". The operator confirms the workpiece type (e.g., plate to plate V-groove, fillet weld, butt weld, and overlay weld). The system then loads the corresponding set of prior parameters (typical weld shape, width-to-depth ratio, desired surface normal range, and typical bevel angle).

[0043] 3. Prior information initialization and quality assessment;

[0044] Based on the selected workpiece type and the initial point cloud quality (point density, noise level, and occlusion level), the system automatically estimates the recognition difficulty score and suggests recognition algorithm categories or parameters (e.g., if the point cloud is flat and unoccluded, geometric fitting is preferred; if the shape is complex or the surface is severely damaged, point cloud classification / learning methods are recommended).

[0045] 4. Interactive parameter adjustment and quick preview;

[0046] The operator can adjust key thresholds (plane fitting threshold, slice thickness, smoothing radius, and RANSAC in-point threshold). The system previews the fitting results of the candidate weld centerline and section in the view in real time (or near real time). After the operator confirms or makes local corrections, they can continue to the next step.

[0047] Step 2: Execute the adaptive recognition algorithm; wherein, based on the obtained prior type, point cloud quality index and annotation information, the recognition module is dynamically selected or combined;

[0048] Based on the obtained prior types, point cloud quality indicators, and operator annotation information, the following recognition modules are dynamically selected or combined:

[0049] 1. RANSAC-based plane intersection module: Suitable for slopes with obvious bevels / slopes (e.g., V-grooves, U-grooves), it fits the normal vectors of the two slopes using the RANSAC algorithm. , Calculate the direction of the intersection line , The centerline position is accurately determined in the neighborhood of the intersection line using the least squares method.

[0050] Randomly sample point sets from both sides of the point cloud of the weld area. , ;

[0051] Fitting two planes using RANSAC: ;in, It is a point in space. , It is the intercept.

[0052] Calculate the direction of the intersection of the two planes : ;in, Normal vectors of the two slopes , The cross product vector; Normal vectors of the two slopes , The magnitude of the cross product vector.

[0053] Solving the system of linear equations yields a point on the intersection line. : ;in, The coefficient matrix, It is a constant vector.

[0054] Optimize the centerline within the intersection neighborhood using weighted least squares: ;in, Represents the function Find the minimum value, which is achieved by adjusting the parameters of the center line. To optimize the target, Indicates curve parameters; For all discrete points in the neighborhood of the intersection line Summation; These are weighting coefficients used to measure the weight of each point. The importance of a point in optimization (e.g., it can be set based on the confidence level of the point, its distance from the center line, etc., and the greater the weight, the stronger its impact on the optimization result). These are discrete points (three-dimensional points in space) within the neighborhood of the intersection line. The centerline to be optimized (with) The curve parameters are represented. It is a point (corresponding centerline parameters). For point To the center line Vector difference at that location.

[0055] Normal unit vector (perpendicular to the center line) The tangent direction at that point is used to describe the direction perpendicular to the centerline, ensuring that optimization is performed within the normal plane. It is the square of the L2 norm (i.e., the square of the length of the vector, used to quantify the magnitude of the bias).

[0056] Near the intersection of the two surfaces, there is a series of discrete points. center line It is the ideal shape of the intersection line to be fitted. Normal direction. Ensure optimization is performed only in the plane perpendicular to the tangent of the intersection line, because the width of the intersection line is normal, and deviations in the tangent direction do not affect the shape of the intersection line.

[0057] Weighted least squares obtains a center line that is statistically the most central and representative of the intersection line by balancing the normal deviations of all points.

[0058] The equation for the output centerline parameters is: ;in, To output the centerline parameters, the positions of all points on the line and the curve parameters are clearly depicted using the form of fixed points + direction vector parameters. The correspondence.

[0059] 2. Cross-section profile analysis module (slice-and-fit): Applicable to V-shaped, U-shaped grooves or obviously symmetrical groove welds. It makes equally spaced slices along the candidate centerline and performs cross-sectional profile fitting (polynomial or symmetrical V / U model) to estimate the width, depth and cross-sectional asymmetry.

[0060] In steps along the centerline Generate slice plane:

[0061] ;in, Indicates the first A slice plane; It is a point in space, or the coordinates of any point on the slice plane; It is the first one on the center line (or reference line) Each point is an anchor point on the center line of the slice plane; The direction of the intersection of the two planes is the normal to the slice plane, because the normal vector of the plane is parallel to the direction vector. A series of parallel planes with step sizes distributed along the center line and perpendicular to the center line are defined by the point normal formula to meet the requirement of generating slices in a step-by-step manner along the direction.

[0062] Extract point cloud within each slice : , For a series of discrete points A set of.

[0063] Fit the contour model in the cross-sectional coordinate system:

[0064] ;

[0065] in, Represents the fitted contour; , and These are the fitting coefficients, which are obtained from the actual contour data using fitting algorithms such as the least squares method. They determine the shape of the fitted curve. For example, the quadratic term in a polynomial is used to characterize the shape features of the contour at the level of a quadratic function, such as the curvature of the contour as expressed in the quadratic term. The fourth-order term in the polynomial is used to capture more complex shape details of the contour, allowing the fitted curve to more accurately fit contours with higher-order variation characteristics. The shape is quantitatively described by mathematically fitting contours with symmetrical properties in a cross-sectional coordinate system.

[0066] After shape fitting, calculate the overall width of the shape. ,depth and symmetry deviation Regarding symmetry deviation The calculation is as follows:

[0067] ;

[0068] in, The area of ​​the left region; The area of ​​the region on the right; The total area of ​​the entire object; Absolute value sign; symmetry deviation In computer vision, symmetry is used to determine the symmetry properties of target objects, assisting in target recognition or defect detection. Symmetry deviation. After calculation, the center points are smoothly connected by continuous slicing to form the center line of the cross section.

[0069] 3. Curvature / ridge extraction module: Applicable to structures with prominent curvature such as fillet welds, weld overlays, and rounded corner surfaces. It extracts ridges or groove centerlines by using point cloud normals and second-order curvature information.

[0070] For estimating the principal curvature of a point cloud:

[0071] ;

[0072] in, and These represent the maximum and minimum curvatures of the point cloud at a point in two mutually perpendicular directions; These are key features that describe local shapes (such as: convex, concave, flat, and sharp). It is a local second-order shape operator (Hessian). Estimating principal curvature for point clouds; providing a quantitative description of the local spatial morphology of 3D point clouds. These two indicators allow for an intuitive understanding of the bending intensity, direction, and overall geometric properties of the point cloud at that point.

[0073] The conditions for finding ridge points are:

[0074] ;

[0075] in, For maximum curvature; Main direction; For the threshold; The gradient of the maximum curvature; The dot product being 0 means and Orthogonal, i.e., the principal curvature is the maximum curvature. The rate of change in its own principal direction is 0, which is the extreme characteristic of the ridge point; in the principal direction, When a local maximum is reached, the gradient has no component along the principal direction. Local linear fitting and spline smoothing are performed on the extracted ridge point set to obtain the centerline trajectory.

[0076] Template Matching / Registration Module (ICP / Local Descriptor): For known standardized weld templates and repetitive workpieces, local descriptor matching is used and ICP fine registration is employed to obtain accurate centerlines.

[0077] Extracting local geometric descriptors (FPFH or SHOT) from a point cloud is as follows:

[0078] ;

[0079] in, Point ( For discrete points within the neighborhood of the intersection line, the local geometric descriptor is a high-dimensional vector (or feature set) used to characterize the local geometry around that point (e.g., flat, sharp, curved, etc.).

[0080] For point The normal vector describes The orientation of the local surface at a point (the direction perpendicular to the local surface);

[0081] The neighborhood radius is used to define the extent of a local region, i.e., by... Centered on, with radius All points within the spherical region constitute The local neighborhood. Representing local geometric descriptors It is by Normal vector and neighborhood radius The geometric features within the defined local region are processed by the function Obtained through mapping. Matched with the template descriptor subset to obtain the initial correspondence. .

[0082] Iterative optimization of rigid transformations using the ICP algorithm:

[0083] ;

[0084] in, This is a rotation matrix used to describe the rotation transformation of the point cloud; is the L2 norm of the Euclidean distance, which measures the distance between two points; It sums the errors of all point pairs; To find the minimum rotation matrix , direction of intersection ; In the target point cloud The corresponding nearest point is determined by searching for the nearest point.

[0085] Represents each point in the source point cloud First, through rigid transformation (rotation matrix) +Direction of intersection ) Obtain the transformed points Then calculate its correspondence with the target point cloud. The square of the Euclidean distance is calculated, and finally the squares of the distances to all points are summed. The ICP algorithm optimizes this through iteration. and This minimizes the distance sum, thus achieving registration (alignment) of the two point clouds. The estimated weld centerline is obtained by transforming the template centerline.

[0086] The engine can run multiple candidate modules in parallel and select the final result or fuse them based on subsequent confidence assessments (e.g., using both planar intersection and cross-sectional analysis methods for the same region, fusing their centerlines and estimating uncertainties).

[0087] Step 3: Output the recognition results and generate the robot arm pose.

[0088] 1. Basic geometry output;

[0089] Output a parametric representation of the weld centerline (3D waypoint sequence or three-dimensional spline curve), including the spatial coordinates (x, y, z), local tangent vector T(s), local normal vector N(s), and curvature κ(s) of each sampling point.

[0090] Provides overall weld parameters: start / end coordinates, total length (arc length along the curve), average / maximum / minimum width, average / maximum / minimum depth, weld type identifier, and confidence score.

[0091] Output segmented width / depth profiles (output cross-sectional measurements along the arc length in steps Δs), and provide abnormal profile markers (e.g., local defects, insufficient depth).

[0092] 2. Generation of welding torch pose (robot-friendly format);

[0093] Based on the centerline tangential T(s) and the local surface / slope normal N(s), calculate the welding torch Cartesian attitude (position + attitude quaternion or Euler angle): The welding torch axis is usually set to be aligned with the bisecting direction of the surface normal or the bevel normal, and the roll angle is set in the tangential direction to control the wire feed direction.

[0094] Simultaneously output the executable pose vector for each sampling point: {position (x,y,z), attitude (α,β,γ), recommended welding process parameters (welding speed, wire diameter, recommended range of welding current), and standoff distance}.

[0095] Perform reachability and collision checks on the generated poses (project the poses back into the robot's inverse kinematics space and check joint limits, singular poses, and collisions between the end effector and the workpiece / fixture), and correct or label poses that are unreachable or at risk of collision.

[0096] 3. Output data format and interface;

[0097] It provides a standardized data interface (JSON / CSV + binary point cloud patch) and is compatible with common industrial robot instructions or offline programming formats (such as mapping examples of trajectory fragment formats such as ABB, KUKA, and Fao).

[0098] Provides a visual report: a 3D view overlaid with a center line, color-coded confidence level, cross-sectional diagram, and key parameter tables, for operators to review and export for archiving.

[0099] 4. Risk warnings and recommended actions;

[0100] If the confidence level is low or an abnormal profile / geometric change is detected, the system will explicitly mark it in the output and provide suggested actions (e.g., resampling, changing the algorithm, manual review, or local rescanning).

[0101] The core of this method has been developed and is running on an autonomous mobile multi-functional welding robot. It has completed weld seam recognition and trajectory simulation on workpieces with straight weld seams. In the future, it will be adapted and made compatible with more types of weld seams and workpieces.

[0102] Example 3;

[0103] Based on Embodiment 2, the modular design of the present invention supports the seamless insertion of new recognition algorithms or new workpiece priors (e.g., adding new deep learning point cloud classification models or new template libraries), and supports long-term online learning to gradually optimize the prior library.

[0104] The method is improved as follows:

[0105] 1. A closed-loop human-machine collaborative startup process is established, consisting of "workpiece type confirmation + manual specification / correction of ROI + prior-based algorithm / parameter automatic recommendation". This step breaks through the traditional binary mode of "fully automatic black box" or "purely manual annotation". Through prior-driven interactive initialization, the false recognition rate is significantly reduced and the algorithm convergence time is shortened, especially improving robustness in complex workpieces with occlusion, noise, or multiple welds.

[0106] 2. A recognition framework of "prior-driven adaptive algorithm selection and multi-method fusion" is proposed. Combining coarse-fine granularity strategies and confidence quantification, the system can automatically select the most suitable recognition path and provide confidence assessment under various working conditions (grooving, fillet weld, complex surface, occlusion, and noise). This design overcomes the shortcomings of poor robustness or adaptability of single methods.

[0107] 3. In addition to outputting geometric information, it also tightly couples weld geometry with welding torch pose, robot accessibility, and process parameters, forming an end-to-end data chain from identification to direct robot execution, including confidence-driven safety checks and executability corrections. This closed-loop output of "identification → pose → accessibility correction → process suggestion" significantly reduces on-site commissioning costs and improves automation deliverability.

[0108] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope described in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An interactive weld seam recognition method based on 3D point clouds, characterized in that, Includes the following steps: Step 1: Perform human-computer interaction; this includes data collection and prior settings. Step 2: Execute the adaptive recognition algorithm; wherein, based on the obtained prior type, point cloud quality index and annotation information, the recognition module is dynamically selected or combined; Step 3: Output the recognition results and generate the robot arm pose.

2. The interactive weld seam recognition method based on three-dimensional point clouds according to claim 1, characterized in that, In step one, the steps of data acquisition and preprocessing, human-computer interaction interface and ROI determination, prior information initialization and quality assessment, and interactive parameter adjustment and quick preview are performed.

3. The interactive weld seam recognition method based on three-dimensional point clouds according to claim 2, characterized in that, In the data acquisition and preprocessing step, an active structured light camera is used to scan the workpiece to obtain a high-density three-dimensional point cloud, which includes x, y and z coordinates as well as intensity / confidence information. The initial processing of the original point cloud includes statistical outlier removal, voxel mesh downsampling, normal vector estimation, and curvature / concavity / convexity calculation; among which, the normal vector estimation is based on neighborhood PCA, with neighborhood radius r or neighboring points k configured.

4. The interactive weld seam recognition method based on three-dimensional point clouds according to claim 2, characterized in that, In the human-computer interaction interface and ROI determination step, based on point cloud visualization on the three-dimensional view, a variety of interactive selection tools are provided to mark or correct the target recognition area.

5. The interactive weld seam recognition method based on three-dimensional point clouds according to claim 1, characterized in that, In step two, the dual-plane intersection fitting module is used for cases with obvious slopes / slopes. It fits the normals of the slopes on both sides using the RANSAC algorithm, calculates the direction of the intersection line, and then uses the least squares method to accurately solve for the centerline position in the neighborhood of the intersection line.

6. The interactive weld seam recognition method based on three-dimensional point clouds according to claim 1, characterized in that, In step two, the cross-sectional profile analysis module is used for V-shaped, U-shaped, or obviously symmetrical bevel welds to make equally spaced slices along the candidate centerline direction and perform cross-sectional profile fitting to estimate the width, depth, and cross-sectional asymmetry.

7. The interactive weld seam recognition method based on three-dimensional point clouds according to claim 1, characterized in that, In step two, the curvature / ridge extraction module is used to extract the ridge line or groove center line of the surface curvature protrusion structure of fillet welds, weld overlays and rounded corners by using point cloud normal and second-order curvature information.

8. The interactive weld seam recognition method based on three-dimensional point clouds according to claim 1, characterized in that, In step two, the template matching / registration module uses local descriptor matching and ICP fine registration to obtain an accurate centerline for known standardized weld templates and repetitive workpieces.

9. The interactive weld seam recognition method based on three-dimensional point clouds according to claim 1, characterized in that, Step three includes steps such as basic geometry output, welding torch pose generation, output data format and interface, and risk warning and suggested actions.

10. An interactive weld recognition system based on three-dimensional point clouds, used to execute the interactive weld recognition method based on three-dimensional point clouds according to any one of claims 1-9, characterized in that, include: The human-computer interaction module is used to perform human-computer interaction. The data processing module, connected to the human-computer interaction module, is used to execute adaptive recognition algorithms; The output module, connected to the data processing module, is used for outputting recognition results and generating the robot arm pose.

Images