A method for robot to autonomously identify and accurately locate welds

Abstract

The present invention proposes a method for robot autonomous identification and precise positioning of welds, which belongs to the field of welding technology. The steps of the method proposed by the present invention include: obtaining a photo of a workpiece to be welded, using a neural network to analyze the photo of the workpiece to be welded to obtain the coordinates of the weld endpoints, scanning the weld to obtain point cloud data according to the coordinates of the weld endpoints, analyzing the point cloud data to automatically calculate the welding gun posture information, performing welding according to the welding gun posture information, and analyzing the position deviation in real time during welding to improve the welding accuracy. Through the steps in the method proposed by the present invention, multiple welds of different types can be identified, the starting and ending points of the welds can be located, the three-dimensional morphology of the welds can be scanned, and the points to be welded can be precisely located without human intervention, and finally the robot's fully autonomous welding can be realized; in addition, the method involved in the present application reuses the camera, so that it plays a role in both weld position identification and weld three-dimensional scanning, thereby reducing costs.

Description

Technical Field

[0001] The invention relates to the technical field of welding, and in particular to a method for a robot to autonomously identify and accurately locate a welding seam. Background Art

[0002] Welding automation is an important means to improve welding quality and efficiency and save labor costs. Weld tracking is one of the necessary technologies to achieve welding automation and has been widely used in the field of high-end equipment manufacturing. However, the current weld tracking technology mainly solves the problem of deviation between the weld teaching position and the actual position, and still cannot solve the problem of positioning the weld initial point. The positioning of the welding initial point and the planning of the weld path still need to rely on offline programming or manual teaching, and it is impossible to achieve full process automation and autonomous welding of robots.

[0003] The traditional method of weld positioning using machine vision can only achieve the welding of planar welds, and cannot achieve the modeling of complex welds or multiple welds. The invention patent application with patent application number 201910266401.4 discloses an intelligent welding system for metal structures, which can also achieve automatic offline programming without teaching. However, the system needs to manually model the workpiece to be welded, and relies on the digital model to provide the robot with the weld position. The welding accuracy is limited by the modeling accuracy and processing accuracy, and the welding path cannot be programmed for complex surfaces that cannot be directly modeled. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present application proposes a method for a robot to autonomously identify and accurately locate welds, so that it can realize the identification of multiple welds of different types, the positioning of the starting and ending points of the welds, the scanning of the three-dimensional topography of the welds, the accurate positioning of the weld points, and the automatic planning of the welding path without human intervention, and finally realize the fully autonomous welding of the robot.

[0005] In order to achieve the above objectives, the present application proposes a method for robot autonomously identifying and accurately locating welds, comprising the following steps:

[0006] Step 1: Use the camera carried by the robot to take a photo of the workpiece to be welded, record the camera's posture information when taking the photo, use the pre-trained neural network to analyze the photo, obtain the number of welds in the photo and the coordinates of the boundary and two endpoints of each weld in the image coordinate system, and calculate the coordinates of the boundary and two endpoints of each weld in the world coordinate system based on the camera's posture information when taking the photo;

[0007] Step 2: Turn on the laser structured light generator and project the structured light onto the workpiece to be welded. For the first weld identified, with the help of the coordinates of its two end points in the world coordinate system calculated in step 1, control the robot to move from the starting point to the end point of the weld at a preset height from the workpiece to be welded, take photos of the structured light during the movement, and record the position information of the camera at the end of the robot when each photo is taken;

[0008] Step 3: Perform image processing on each structured light photo taken in step 2 to obtain the coordinates of the center line of each structured light stripe in the image coordinate system, and then calculate the coordinates of the center line of the structured light stripe in the world coordinate system in combination with the pose information of the robot end camera corresponding to the structured light photo, and combine the coordinates of the center line of all structured light stripes in the world coordinate system into a set of point cloud data;

[0009] Step 4: Combined with the weld boundary determined in step 1, the point cloud data outside the weld boundary is removed to reduce the data volume, and the position information of the welding gun on the robot is obtained by analyzing the remaining point cloud data after removal;

[0010] Step 5: Repeat steps 2 to 4 until the welding gun posture information corresponding to all welds is obtained;

[0011] Step 6. Control the robot to perform welding operations on each weld in turn according to the welding gun posture information. During the welding process, turn on the laser structured light generator, project the structured light onto the workpiece to be welded, take a photo of the structured light, and obtain the precise coordinates of the weld by analyzing the photo of the structured light. At the same time, monitor the deviation between the current position of the end of the welding gun and the precise coordinates of the weld, and control the robot to move in the direction of reducing the deviation.

[0012] In some embodiments, in step 1, the specific method of using a pre-trained neural network to analyze the photo and obtain the number of welds in the photo and the coordinates of the boundary and two end points of each weld in the image coordinate system is as follows:

[0013] Step 11, constructing a data set, the data set includes photos of plane welds, curved welds, straight welds, curved welds, single welds, multiple welds, and cross welds;

[0014] Step 12, annotate each photo in the data set constructed in step 11, and confirm one by one whether the pixel points in the photo belong to the weld, the workpiece to be welded, or the background. The pixel points belonging to the weld are marked as 1, the pixel points belonging to the workpiece to be welded are marked as 2, and the pixel points belonging to the background are marked as 0;

[0015] Step 13: construct a semantic segmentation neural network, and then use the data set labeled in step 12 to train the semantic segmentation neural network to obtain a trained neural network;

[0016] Step 14, input the photo of the workpiece to be welded into the neural network trained in step 13, the neural network will output the category label of each pixel in the photo, select all the pixel points marked as 1 that belong to the weld, and in the area composed of them, each connected domain with an area greater than the preset value corresponds to a weld in the input photo, the number of connected domains with an area greater than the preset value is the number of welds in the photo, the pixel coordinates of the boundary of the connected domain with an area greater than the preset value is the coordinates of the boundary of the weld in the image coordinate system, and the pixel coordinates of the endpoints of the connected domain with an area greater than the preset value is the coordinates of the endpoints of the weld in the image coordinate system.

[0017] In some embodiments, in step 3, the image processing process is to perform filtering, noise reduction, morphological operation, edge detection, and extraction of the center line of the structured light stripes on the structured light photo in sequence.

[0018] In some embodiments, in step 4, the specific method of obtaining the position information of the welding gun on the robot by analyzing the remaining point cloud data after elimination is as follows:

[0019] Step 41, performing plane or surface fitting on the point cloud data, the points on the fitting surface belong to the surface of the workpiece to be welded, and the points outside the fitting surface belong to the weld;

[0020] Step 42, dividing the point cloud data consisting of points belonging to the surface of the workpiece to be welded into two parts along the weld, and recording them as a first group of point cloud data of the surface of the workpiece to be welded and a second group of point cloud data of the surface of the workpiece to be welded;

[0021] Step 43, the point cloud data consisting of points belonging to the weld is divided into several groups along the extension direction of the weld and the centroid points are extracted respectively;

[0022] Step 44, for any extracted centroid point, the coordinates of the centroid point are the position of the welding gun in the welding gun posture information corresponding to the centroid point, and the direction of the line connecting the two centroid points before and after the centroid point is the x direction of the welding gun posture in the welding gun posture information corresponding to the centroid point; extract all points whose distance from the centroid point is less than k from the first group of surface point cloud data of the workpiece to be welded to form a new group of point cloud data, extract all points whose distance from the centroid point is less than k from the second group of surface point cloud data of the workpiece to be welded to form a new group of point cloud data, k is a preset value, respectively obtain the normal vectors of the plane or approximate plane formed by the two groups of new point cloud data, obtain the first normal vector and the second normal vector, and obtain the third normal vector by summing the first normal vector and the second normal vector. The direction indicated by the third normal vector is the z direction of the welding gun posture in the welding gun posture information corresponding to the centroid point. When the x direction and the z direction are determined, the y direction of the welding gun posture in the welding gun posture information corresponding to the centroid point can be obtained by the right-hand rule.

[0023] In some embodiments, in step 6, the steps used to analyze the structured light photograph are filtering, denoising, morphological operations, edge detection, extracting the center line of the structured light stripes, extracting the feature points of the weld on the center line, and calculating the precise coordinates of the weld.

[0024] The beneficial effect of the scheme of the present application is that the above-mentioned method of autonomously identifying and accurately locating welds by the robot can realize the identification of multiple welds of different types, the positioning of the starting and ending points of the welds, the scanning of the three-dimensional morphology of the welds, the precise positioning of the weld points, and the automatic planning of the welding path without human intervention, and finally realize the fully autonomous welding of the robot; the core idea of ​​the scheme of the present application is to use powerful neural network technology to analyze the photos of the workpiece to be welded so as to obtain the position and direction information of the welds, so as to provide guidance for the subsequent three-dimensional scanning of the workpiece to be welded. This scheme can reduce the workload of three-dimensional scanning and the complexity of data processing after scanning, and can significantly improve the execution speed of the algorithm; in addition, the method involved in the present application reuses the camera, so that it plays a role in both weld position identification and weld three-dimensional scanning, thereby reducing costs. The fully autonomous identification and positioning technology of welds proposed in the present application can provide support for the realization of a wide range of intelligent manufacturing. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flow chart of a method for a robot to autonomously identify and accurately locate a weld in an embodiment is shown.

[0026] Figure 2 Schematic diagrams of a linear weld and its boundary in an embodiment are shown, wherein (a) is a top view schematic diagram and (b) is a front view schematic diagram.

[0027] Figure 3 Schematic diagrams of an S-shaped weld and its boundary in an embodiment are shown, wherein (a) is a top view schematic diagram and (b) is a front view schematic diagram.

[0028] Figure 4 The figure shows a structured light photo taken in step 2 and the process diagram of analyzing the structured light photo in step 3, where (a) is the original image, (b) is the image after filtering, (c) is the image after edge detection, and (d) is the image after centerline extraction.

[0029] Figure 5 A visualization of the point cloud data generated in step three is shown.

[0030] Figure 6 A visualization of the point cloud data consisting of points belonging to the weld in step 4 is shown.

[0031] Figure 7 The deviation diagram between the precise coordinates of the weld obtained in step six and the current position of the welding gun tip is shown. DETAILED DESCRIPTION

[0032] The specific implementation of the present application is further described below in conjunction with the accompanying drawings.

[0033] like Figure 1 As shown, the method for robot autonomous identification and precise positioning of welds involved in the present application comprises the following steps:

[0034] Step 1: Use the camera carried by the robot to take a photo of the workpiece to be welded, record the camera's posture information when taking the photo, use a pre-trained neural network to analyze the photo, obtain the number of welds in the photo (for example, n) and the coordinates of the boundary and two endpoints of each weld in the image coordinate system, and calculate the coordinates of the boundary and two endpoints of each weld in the world coordinate system in combination with the camera's posture information when taking the photo.

[0035] In this embodiment, the robot used is Fanuc M-20iB / 25, and the camera is The ImagingSource DFK23GM021 (German ImagingSource industrial camera). Specifically, the camera carried by the robot is calibrated, and the robot and the camera are calibrated by hand and eye. The neural network involved in this step can use the YOLOv5 neural network, which has the advantage of fast recognition speed and can improve the recognition efficiency of welds. In this embodiment, the workpiece to be welded is a straight or S-shaped butt weld with a V-shaped groove, such as Figure 2-3 As shown, in the figure, 1 represents the workpiece to be welded, and 2 represents the weld.

[0036] In this step, the pre-trained neural network is used to analyze the photo to obtain the number of welds in the photo and the coordinates of the boundary and two end points of each weld in the image coordinate system. The specific steps are as follows:

[0037] Step 11: construct a data set, which includes photos of plane welds, curved welds, straight welds, curved welds, single welds, multiple welds, and cross welds.

[0038] Step 12: Label each photo in the data set constructed in step 11, and confirm one by one whether the pixels in the photo belong to the weld, the workpiece to be welded, or the background. The pixels belonging to the weld are marked as 1, the pixels belonging to the workpiece to be welded are marked as 2, and the pixels belonging to the background are marked as 0.

[0039] Step 13: construct a semantic segmentation neural network, and then use the data set annotated in step 12 to train the semantic segmentation neural network to obtain a trained neural network. In this step, the semantic segmentation neural network uses a variant of U-Net. Optionally, the semantic segmentation neural network can also use a neural network such as SegNet, PSPNet, or a variant thereof.

[0040] Step 14, input the photo of the workpiece to be welded into the neural network trained in step 13, the neural network will output the category label of each pixel in the photo, select all the pixel points marked as 1 that belong to the weld, and in the area composed of them, each connected domain with an area greater than the preset value corresponds to a weld in the input photo, the number of connected domains with an area greater than the preset value is the number of welds in the photo, the pixel coordinates of the boundary of the connected domain with an area greater than the preset value is the coordinates of the boundary of the weld in the image coordinate system, and the pixel coordinates of the endpoints of the connected domain with an area greater than the preset value is the coordinates of the endpoints of the weld in the image coordinate system.

[0041] Step 2: Turn on the laser structured light generator and project the structured light onto the workpiece to be welded. For the first weld identified, use the coordinates of its two endpoints in the world coordinate system calculated in step 1 to control the robot to move from the starting point (one of the endpoints) of the weld to the end point (the other endpoint) at a preset height from the workpiece to be welded. During the movement, take photos of the structured light and record the position information of the camera at the end of the robot when each photo is taken.

[0042] In this embodiment, the laser structured light generator adopts a blue light inline laser with a wavelength of 450nm.

[0043] Step 3: Perform image processing on each structured light photo taken in step 2 to obtain the coordinates of the center line of each structured light stripe in the image coordinate system, and then calculate the coordinates of the center line of the structured light stripe in the world coordinate system in combination with the pose information of the robot end camera corresponding to the structured light photo. Combine the coordinates of the center line of all structured light stripes in the world coordinate system into a set of point cloud data, such as Figure 5 shown.

[0044] In this step, specifically, the image processing process may be filtering, noise reduction, morphological operation, edge detection, and extraction of the center line of the structured light stripes in sequence on the structured light photo. Figure 4 shown.

[0045] Step 4: Combined with the weld boundary determined in step 1, the point cloud data outside the weld boundary is eliminated to reduce the data volume, and the position information of the welding gun on the robot is obtained by analyzing the remaining point cloud data after elimination.

[0046] In this step, the specific method of obtaining the posture information of the welding gun on the robot by analyzing the remaining point cloud data after elimination is as follows:

[0047] Step 41, plane or surface fitting is performed on the point cloud data, the points on the fitting surface belong to the surface of the workpiece to be welded, and the points outside the fitting surface belong to the weld.

[0048] Step 42, the point cloud data consisting of points belonging to the surface of the workpiece to be welded is divided into two parts along the weld, which are respectively recorded as a first group of point cloud data of the surface of the workpiece to be welded and a second group of point cloud data of the surface of the workpiece to be welded.

[0049] Step 43: The point cloud data consisting of points belonging to the weld is divided into several groups along the extension direction of the weld and the centroid points are extracted respectively.

[0050] Step 44, for any extracted centroid point, the coordinates of the centroid point are the position of the welding gun in the welding gun posture information corresponding to the centroid point, and the direction of the line connecting the two centroid points before and after the centroid point is the x direction of the welding gun posture in the welding gun posture information corresponding to the centroid point; extract all points whose distance from the centroid point is less than k from the first group of surface point cloud data of the workpiece to be welded to form a new group of point cloud data, extract all points whose distance from the centroid point is less than k from the second group of surface point cloud data of the workpiece to be welded to form a new group of point cloud data, k is a preset value, which is set according to actual needs, and respectively obtain the normal vectors of the plane or approximate plane formed by the two groups of new point cloud data to obtain the first normal vector and the second normal vector, and obtain the third normal vector by summing the first normal vector and the second normal vector, and the direction indicated by the third normal vector is the z direction of the welding gun posture in the welding gun posture information corresponding to the centroid point, and when the x direction and the z direction are determined, the y direction of the welding gun posture in the welding gun posture information corresponding to the centroid point can be obtained by the right-hand rule.

[0051] Step 5: Repeat steps 2 to 4 until the welding gun posture information corresponding to all welds is obtained.

[0052] Step 6: Control the robot to perform welding operations on each weld in turn according to the welding gun posture information. During the welding process, turn on the laser structured light generator, project the structured light onto the workpiece to be welded, take photos of the structured light, and obtain the precise coordinates of the weld by analyzing the photos of the structured light. At the same time, monitor the deviation between the current position of the welding gun end and the precise coordinates of the weld, and control the robot to move in the direction of reducing the deviation, such as Figure 7 shown.

[0053] In this step, the steps used to analyze the structured light photo are filtering, noise reduction, morphological operation, edge detection, extracting the center line of the structured light stripes, extracting the characteristic points of the weld on the center line, and calculating the precise coordinates of the weld.

[0054] The method for autonomously identifying and accurately locating welds by robots involved in this application can solve the following problems:

[0055] 1) Step 1 can solve the problem of simultaneous identification of multiple welds and autonomous positioning of the initial points of welds.

[0056] 2) Steps 2-4 can solve the problem of automatic welding path planning for complex curved surface welds.

[0057] 3) Step 6 can solve the problem that the pre-set welding gun posture caused by welding deformation does not match the actual position of the weld.

[0058] The method of autonomous robot identification and precise positioning of welds involved in the present application can realize the identification of multiple welds of different types, the positioning of the starting and ending points of welds, the scanning of the three-dimensional morphology of welds, the precise positioning of the points to be welded, and the automatic planning of welding paths without human intervention, and finally realize the fully autonomous welding of robots. The core idea of ​​the scheme of the present application is to use powerful neural network technology to analyze the photos of the workpiece to be welded to obtain the position and direction information of the weld, so as to provide guidance for the subsequent three-dimensional scanning of the workpiece to be welded. This scheme can reduce the workload of three-dimensional scanning and the complexity of data processing after scanning, and can significantly improve the execution speed of the algorithm; in addition, the method involved in the present application reuses the camera, so that it plays a role in both weld position identification and weld three-dimensional scanning, reducing costs. The fully autonomous identification and positioning technology of welds proposed in the present application can provide support for the realization of a wide range of intelligent manufacturing.

[0059] The above is only a preferred specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can make equivalent substitutions or changes according to the technical solution and concept of the present application within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A method for autonomously identifying and accurately locating welds by a robot, characterized in that: The following steps are involved: Step 1: Use the camera carried by the robot to take a photo of the workpiece to be welded, record the camera's posture information when taking the photo, use the pre-trained neural network to analyze the photo, obtain the number of welds in the photo and the coordinates of the boundary and two endpoints of each weld in the image coordinate system, and calculate the coordinates of the boundary and two endpoints of each weld in the world coordinate system based on the camera's posture information when taking the photo; Step 2: Turn on the laser structured light generator and project the structured light onto the workpiece to be welded. For the first weld identified, with the help of the coordinates of its two end points in the world coordinate system calculated in step 1, control the robot to move from the starting point to the end point of the weld at a preset height from the workpiece to be welded, take photos of the structured light during the movement, and record the position information of the camera at the end of the robot when each photo is taken; Step 3: Perform image processing on each structured light photo taken in step 2 to obtain the coordinates of the center line of each structured light stripe in the image coordinate system, and then calculate the coordinates of the center line of the structured light stripe in the world coordinate system in combination with the pose information of the robot end camera corresponding to the structured light photo, and combine the coordinates of the center line of all structured light stripes in the world coordinate system into a set of point cloud data; Step 4: Combined with the weld boundary determined in step 1, the point cloud data outside the weld boundary is removed to reduce the data volume, and the position information of the welding gun on the robot is obtained by analyzing the remaining point cloud data after removal; Step 5: Repeat steps 2 to 4 until the welding gun posture information corresponding to all welds is obtained; Step 6. Control the robot to perform welding operations on each weld in turn according to the welding gun posture information. During the welding process, turn on the laser structured light generator, project the structured light onto the workpiece to be welded, take a photo of the structured light, and obtain the precise coordinates of the weld by analyzing the photo of the structured light. At the same time, monitor the deviation between the current position of the end of the welding gun and the precise coordinates of the weld, and control the robot to move in the direction of reducing the deviation.

2. The method for autonomously identifying and accurately locating welds by a robot according to claim 1, characterized in that: In step 1, the specific method of using a pre-trained neural network to analyze the photo and obtain the number of welds in the photo and the coordinates of the boundary and two end points of each weld in the image coordinate system is as follows: Step 11, constructing a data set, the data set includes photos of plane welds, curved welds, straight welds, curved welds, single welds, multiple welds, and cross welds; Step 12, annotate each photo in the data set constructed in step 11, and confirm one by one whether the pixel points in the photo belong to the weld, the workpiece to be welded, or the background. The pixel points belonging to the weld are marked as 1, the pixel points belonging to the workpiece to be welded are marked as 2, and the pixel points belonging to the background are marked as 0; Step 13: construct a semantic segmentation neural network, and then use the data set labeled in step 12 to train the semantic segmentation neural network to obtain a trained neural network; Step 14, input the photo of the workpiece to be welded into the neural network trained in step 13, the neural network will output the category label of each pixel in the photo, select all the pixel points marked as 1 that belong to the weld, and in the area composed of them, each connected domain with an area greater than the preset value corresponds to a weld in the input photo, the number of connected domains with an area greater than the preset value is the number of welds in the photo, the pixel coordinates of the boundary of the connected domain with an area greater than the preset value is the coordinates of the boundary of the weld in the image coordinate system, and the pixel coordinates of the endpoints of the connected domain with an area greater than the preset value is the coordinates of the endpoints of the weld in the image coordinate system.

3. The method for autonomously identifying and accurately locating welds by a robot according to claim 1, characterized in that: In step 3, the image processing process is to perform filtering, noise reduction, morphological operation, edge detection, and extraction of the center line of the structured light stripes on the structured light photo in sequence.

4. The method for autonomously identifying and accurately locating welds by a robot according to claim 1, characterized in that: In step 4, the specific method of obtaining the position information of the welding gun on the robot by analyzing the remaining point cloud data after elimination is as follows: Step 41, performing plane or surface fitting on the point cloud data, the points on the fitting surface belong to the surface of the workpiece to be welded, and the points outside the fitting surface belong to the weld; Step 42, dividing the point cloud data consisting of points belonging to the surface of the workpiece to be welded into two parts along the weld, and recording them as a first group of point cloud data of the surface of the workpiece to be welded and a second group of point cloud data of the surface of the workpiece to be welded; Step 43, the point cloud data consisting of points belonging to the weld is divided into several groups along the extension direction of the weld and the centroid points are extracted respectively; Step 44, for any extracted centroid point, the coordinates of the centroid point are the position of the welding gun in the welding gun posture information corresponding to the centroid point, and the direction of the line connecting the two centroid points before and after the centroid point is the x direction of the welding gun posture in the welding gun posture information corresponding to the centroid point; extract all points whose distance from the centroid point is less than k from the first group of surface point cloud data of the workpiece to be welded to form a new group of point cloud data, extract all points whose distance from the centroid point is less than k from the second group of surface point cloud data of the workpiece to be welded to form a new group of point cloud data, k is a preset value, respectively obtain the normal vectors of the plane or approximate plane formed by the two groups of new point cloud data, obtain the first normal vector and the second normal vector, and obtain the third normal vector by summing the first normal vector and the second normal vector. The direction indicated by the third normal vector is the z direction of the welding gun posture in the welding gun posture information corresponding to the centroid point. When the x direction and the z direction are determined, the y direction of the welding gun posture in the welding gun posture information corresponding to the centroid point can be obtained by the right-hand rule.

5. The method for autonomously identifying and accurately locating welds by a robot according to claim 1, characterized in that: In step 6, the steps used to analyze the structured light photo are filtering, noise reduction, morphological operation, edge detection, extracting the center line of the structured light stripes, extracting the characteristic points of the weld on the center line, and calculating the precise coordinates of the weld.

Images