Welding robot welding seam positioning method and system based on pose estimation

Abstract

The invention relates to a welding robot welding seam positioning method and system based on pose estimation. The method comprises the steps that hand-eye calibration is conducted on an RGB camera and a structured light camera; calibrating internal parameters of the RGB camera; training a steel member pose estimation deep neural network; driving an RGB camera to acquire an image and estimating the pose of the steel member; adjusting the posture of the welding robot based on the posture of the steel component so as to adjust the sampling posture of a structured light camera fixed at the tail end of the welding robot; and the structured light camera is driven to collect three-dimensional point cloud data of the target welding area, and a welding seam is extracted from the three-dimensional point cloud data. Compared with the prior art, the advantages that the RGB camera is wide in view field and the structured light camera is high in precision are combined, the defect that an existing welding seam positioning technology is poor in flexibility due to the fact that the range of the structured light camera is limited can be overcome, and the welding robot welding seam positioning process is more intelligent and efficient.

Description

Technical Field

[0001] The present invention relates to the technical field of welding robot control, and particularly to a welding robot weld seam positioning method and system based on pose estimation. Background Art

[0002] Welding robots are gradually replacing welders to complete welding operations. Weld seam positioning is the basis for ensuring the quality of welding robot operations.

[0003] Currently, welding robot weld seam positioning technology mainly uses a structured light camera to collect three-dimensional point cloud data of the target welding area, and further analyzes the three-dimensional point cloud data to extract the weld seam. For example, an adaptive welding system disclosed in the invention with the publication number CN116748742B includes the following: a laser vision sensor is connected to the welding robot and the laser vision sensor is arranged on the welding device; the laser vision sensor is used to obtain the geometric information of the welding workpiece, and determine the welding parameters by looking up a table according to the geometric information, and send the welding parameters to the welding robot; the welding parameters include welding process parameters and position signals; the welding robot is connected to the welding device; the welding robot is used to adjust the position of the welding device relative to the welding workpiece according to the position signal, and send the welding process parameters to the welding device; the welding device welds the welded part according to the welding process parameters.

[0004] Although the structured light (laser) camera has high sampling accuracy, its limited field of view makes it often impossible to collect three-dimensional point cloud data of the target welding area in complex welding scenarios because the steel member is not placed in the specified position, resulting in weld seam positioning failure. In the actual steel member production scenario, there is a certain difference between the actual placement position and the ideal placement position of the steel member, and it is necessary to manually adjust the posture of the robotic arm to ensure that the structured light camera fixed at the end of the robotic arm accurately aligns with the target welding position, which will obviously reduce the automation level of the welding robot and limit the application range of the welding robot. Summary of the Invention

[0005] The purpose of the present invention is to overcome the above-mentioned defects of the existing technology that the structured light camera cannot collect three-dimensional point cloud data of the target welding area, resulting in weld seam positioning failure and the need for manual adjustment, and to provide a welding robot weld seam positioning method and system based on pose estimation.

[0006] The purpose of the present invention can be achieved by the following technical solutions: A welding robot weld seam positioning method based on pose estimation includes the following steps: Fix the RGB camera in advance and fix the structured light camera at the end of the robotic arm of the welding robot; Perform hand-eye calibration on the RGB camera and the structured light camera to obtain the transformation relationships between the RGB camera coordinate system and the end-effector coordinate system (or the robotic arm coordinate system), and between the structured light camera coordinate system and the end-effector coordinate system; Perform internal parameter calibration on the RGB camera; Drive the RGB camera to collect images through the welding robot, and estimate the pose of the steel structure through the pre-constructed and trained deep neural network for steel component pose estimation and the internal parameter calibration results of the RGB camera; Adjust the pose of the welding robot based on the pose of the steel structure, as well as the transformation relationships between the RGB camera coordinate system and the end-effector coordinate system (or the robotic arm coordinate system), and between the structured light camera coordinate system and the end-effector coordinate system, so as to adjust the sampling pose of the structured light camera fixed on the welding robot; Drive the structured light camera to collect three-dimensional point cloud data of the target welding area, and extract the weld seam from the three-dimensional point cloud data for the welding robot to perform welding drive control.

[0007] Furthermore, the adjustment process of the sampling pose of the structured light camera is specifically as follows: Preset the sampling pose of the structured light camera in the steel component coordinate system; Transform the pose of the steel structure in the RGB camera coordinate system to the end-effector coordinate system; Based on the preset sampling pose of the structured light camera in the steel component coordinate system and the pose of the steel structure in the end-effector coordinate system, calculate the sampling pose of the structured light camera in the end-effector coordinate system; Based on the sampling pose of the structured light camera in the end-effector coordinate system, adjust the pose of the welding robot to drive the structured light camera to adjust its pose.

[0008] Furthermore, the preset sampling pose of the structured light camera in the steel component coordinate system is set based on the range of the structured light camera and the physical dimensions of the structured light camera, the welding torch, and the steel component.

[0009] Furthermore, the training process of the deep neural network for steel component pose estimation includes: Construct the data required for training the deep neural network for steel component pose estimation to train the deep neural network for steel component pose estimation. The deep neural network for steel component pose estimation obtains the pose of the steel structure by analyzing the image features of the data during the training process.

[0010] Furthermore, the data required for training the deep neural network includes a real training data set and a virtual training data set.

[0011] Furthermore, driving the RGB camera to collect images to estimate the pose of the steel structure includes the cases of eye-in-hand and eye-to-hand.

[0012] Further, the process of extracting the weld seam from the three-dimensional point cloud data is specifically as follows: Downsample the three-dimensional point cloud data, perform plane segmentation on the downsampled three-dimensional point cloud data, and obtain the plane intersection line; Based on the pose of the steel structure and the pre-obtained steel structure model, obtain the theoretical position of the target weld seam; Based on the distance between the plane intersection line and the theoretical position of the target weld seam, obtain the true three-dimensional coordinates of the target weld seam.

[0013] Further, the deep neural network for steel component pose estimation is a convolutional neural network or a residual neural network.

[0014] Further, during the hand-eye calibration process, fix the calibration board at the same position, drive the welding robot to collect multiple images containing the calibration board with different poses of the RGB camera and the structured light camera, and then perform hand-eye calibration processing.

[0015] The present invention also provides a welding robot weld seam positioning system based on pose estimation, including a memory and a processor. The memory stores a computer program, and the processor calls the computer program to execute the steps of the method described above.

[0016] Compared with the prior art, the present invention has the following advantages: (1) The welding robot weld seam positioning method based on pose estimation provided by the present invention uses an RGB camera combined with a deep neural network to obtain the pose of the steel component within a large field of view, and adjusts the pose of the welding robot based on the pose of the steel component to adjust the sampling pose of the structured light camera fixed at the end of the welding robot, so that the structured light camera can collect the three-dimensional point cloud data of the target welding area; further analyze the three-dimensional point cloud data to extract the three-dimensional information of the weld seam. By introducing pose estimation into the welding robot weld seam positioning, combining the advantages of the wide field of view of the RGB camera and the high accuracy of the structured light camera, the problem that it is difficult for the welding robot to automatically locate the weld seam in a complex scene due to the short range of the structured light camera is solved on the premise of ensuring the weld seam positioning accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic structural diagram of the present invention Figure 1 It is a schematic flow diagram of a welding robot weld seam positioning method based on pose estimation proposed in an embodiment of the present invention; Figure 2 It is a schematic diagram of obtaining the pose of a steel component based on an RGB camera in an embodiment of the present invention; Figure 3It is a schematic diagram of optimizing the sampling pose of a structured light camera fixed on a robotic arm by changing the pose of a steel member in an embodiment of the present invention; Figure 4 It is a schematic diagram of the sampling pose of a preset structured light camera in the coordinate system of a steel member in an embodiment of the present invention; Figure 5 It is a schematic diagram of the process change of extracting a weld seam from a three-dimensional point cloud in an embodiment of the present invention. Detailed implementation manners

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. Generally, the components of the embodiments of the present invention described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations.

[0019] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed present invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0020] It should be noted that: similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0021] Embodiment 1 As Figure 1 shown, this embodiment provides a weld seam positioning method for a welding robot based on pose estimation, including the following steps: S1: Fix an RGB camera and a structured light camera at the end of the robotic arm of the welding robot in advance; perform hand-eye calibration on the RGB camera and the structured light camera to obtain the transformation relationship between the RGB camera coordinate system, the structured light camera coordinate system, and the end coordinate system of the robotic arm; S2: Perform internal parameter calibration on the RGB camera; S3: Train a deep neural network for steel member pose estimation; S4: Drive the RGB camera to collect images by the welding robot, and estimate the pose of the steel structure through the pre-constructed and trained deep neural network for steel member pose estimation and the internal parameter calibration result of the RGB camera; S5: Adjust the pose of the welding robot based on the pose of the steel structure, as well as the transformation relationships among the RGB camera coordinate system, the structured light camera coordinate system, and the end - effector coordinate system of the robotic arm, so as to adjust the sampling pose of the structured light camera fixed on the welding robot. S6: Drive the structured light camera to collect the three - dimensional point cloud data of the target welding area, and extract the weld seam from the three - dimensional point cloud data for the welding robot to perform welding drive control.

[0022] The following is a specific description of each step.

[0023] In this embodiment, the process of step S1 for performing hand - eye calibration on the RGB camera and the structured light camera includes: Fix the RGB camera and the structured light camera on the end - effector of the robotic arm of the welding robot. Fix the calibration board at the same position, drive the robot to enable the RGB camera and the structured light camera to collect multiple images containing the calibration board in different poses, and use the hand - eye calibration algorithm to calculate the transformation relationships among the RGB camera coordinate system, the structured light camera coordinate system, and the end - effector coordinate system of the robot.

[0024] In this embodiment, step S2 is specifically: The internal parameter calibration of the RGB camera includes placing the calibration board in different poses at different positions in the field of view of the RGB camera, and using the RGB camera to collect multiple images containing the calibration board, and using the camera internal parameter calibration algorithm to calibrate the internal parameters of the RGB camera.

[0025] In this embodiment, step S3 is specifically: Training the steel component pose estimation deep neural network includes constructing a steel component pose estimation training dataset and training the deep neural network on a processor. The steel component pose estimation deep neural network analyzes the features of the steel component RGB image to obtain the pose of the steel component and marks it with a bounding box.

[0026] As an alternative implementation, the steel component pose estimation training dataset includes a real training dataset and a virtual training dataset.

[0027] As an alternative implementation, the steel component pose estimation deep neural network includes, but is not limited to, networks such as convolutional neural networks and residual neural networks.

[0028] In this embodiment, step S4 is specifically: Refer to Figure 2 , driving the RGB camera to collect images and estimate the pose of the steel component includes using the RGB camera to collect clear and complete pictures of the steel component, and using the trained deep neural network to analyze the image features to obtain the pose of the steel component in the camera coordinate system. It can be understood that the deep neural network is the deep neural network trained in step S3.

[0029] Driving an RGB camera to collect images for estimating the pose of a steel structure includes the cases where the RGB camera is fixed at the end of the robotic arm of a welding robot or fixed in space.

[0030] In this embodiment, step S5 is specifically as follows: Refer to Figure 3 , adjusting the pose of the welding robot based on the pose of the steel member to adjust the sampling pose of the structured light camera fixed at the end of the welding robot includes presetting the sampling pose of the structured light camera in the coordinate system of the steel member; Refer to Figure 4 , the sampling pose of the structured light camera should fully consider the range of the structured light camera, as well as the physical dimensions of the camera, welding torch, workpiece, and other objects, and enable the sampling field of view of the structured light camera to cover the target welding area without collision; In this embodiment, adjusting the pose of the welding robot based on the pose of the steel member to adjust the sampling pose of the structured light camera fixed at the end of the welding robot further includes adjusting the pose of the steel member from the camera coordinate system to the robot coordinate system based on the transformation relationship between the RGB camera coordinate system and the robot coordinate system, and adjusting the structured light sampling pose in the robot coordinate system based on the preset structured light sampling pose and the pose of the steel member in the robot coordinate system; It can be understood that the transformation relationship between the RGB camera coordinate system and the robot coordinate system can be calculated from the transformation relationship between the RGB camera coordinate system and the robot end coordinate system obtained by calibration in step S1 and the robot end pose, and the robot end pose can be directly obtained from the robot control system.

[0031] In this embodiment, in step S6, exemplarily, the three-dimensional point cloud data of the target welding area collected includes controlling the structured light camera to project a known grating pattern onto the target area, and calculating the three-dimensional point cloud data of the target area by capturing the deformation of these patterns by the camera.

[0032] Exemplarily, extracting the weld seam from the three-dimensional point cloud data includes transferring the three-dimensional point cloud data of the target welding area from the structured light camera coordinate system to the robot coordinate system based on the transformation relationship between the structured light camera coordinate system and the robot coordinate system. It can be understood that the transformation relationship between the structured light camera coordinate system and the robot coordinate system can be calculated from the transformation relationship between the structured light camera coordinate system and the robot end coordinate system obtained by calibration in step S1 and the robot end pose, and the robot end pose can be directly obtained from the robot control system.

[0033] Exemplarily, refer to Figure 5 , extracting the weld seam from the three-dimensional point cloud data further includes downsampling the three-dimensional point cloud of the target welding area, and performing plane segmentation on the three-dimensional point cloud and extracting the plane intersection line. Figure 5The gray line is the intersection line of the planes. Further, based on the pose of the steel component and the CAD model of the steel component, the theoretical position of the weld seam in the robot coordinate system is obtained, and based on the distance between the intersection line of the planes and the theoretical position of the weld seam, the true three-dimensional coordinates of the target weld seam are obtained. The intersection line with the smallest distance from the theoretical weld seam position is the true weld seam position. Figure 5 The green line on the right is the theoretical position of the weld seam in the robot coordinate system. Figure 5 The blue line on the right is the true three-dimensional coordinates of the target weld seam. It can be understood that the pose of the steel component is the pose of the steel component obtained by the deep neural network based on the RGB image in step S5, and the CAD model of the steel component can be directly obtained in the field of steel component processing.

[0034] The present invention also provides a welding robot weld seam positioning system based on pose estimation, including a memory and a processor. The memory stores a computer program, and the processor calls the computer program to execute the steps of the above welding robot weld seam positioning method based on pose estimation.

[0035] The preferred specific embodiments of the present invention have been described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and variations according to the concept of the present invention without creative labor. Therefore, all technical solutions that can be obtained by those skilled in the art in the technical field of the present invention based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the protection scope determined by the claims.

Claims

1. A welding robot weld positioning method based on posture estimation, characterized in that: The following steps are involved: Fix the RGB camera in advance, and fix the structured light camera on the end of the robotic arm of the welding robot; Perform hand-eye calibration on the RGB camera and structured light camera to obtain the transformation relationship between the RGB camera coordinate system and the robotic arm end coordinate system, and between the structured light camera coordinate system and the robotic arm end coordinate system; Perform intrinsic calibration on the RGB camera; The welding robot drives the RGB camera to collect images, and estimates the steel structure pose through the pre-built and trained deep neural network for steel component pose estimation and the intrinsic calibration results of the RGB camera. The posture of the welding robot is adjusted based on the posture of the steel structure, the coordinate system of the RGB camera and the coordinate system of the end of the robot arm, and the coordinate system of the structured light camera and the coordinate system of the end of the robot arm, so as to adjust the sampling posture of the structured light camera fixed on the welding robot; The structured light camera is driven to collect three-dimensional point cloud data of the target welding area, and the weld is extracted from the three-dimensional point cloud data for use in the welding drive control of the welding robot.

2. A welding robot weld positioning method based on posture estimation according to claim 1, characterized in that: The adjustment process of the sampling posture of the structured light camera is specifically as follows: Preset the sampling pose of the structured light camera in the steel component coordinate system; Transform the steel structure pose in the RGB camera coordinate system to the robot end coordinate system; Based on the preset sampling pose of the structured light camera in the steel component coordinate system and the steel structure pose in the robot terminal coordinate system, the sampling pose of the structured light camera in the robot terminal coordinate system is calculated; Based on the sampling pose of the structured light camera in the robot terminal coordinate system, the pose of the welding robot is adjusted to drive the structured light camera to adjust its pose.

3. A welding robot weld positioning method based on posture estimation according to claim 2, characterized in that: The preset sampling pose of the structured light camera in the steel component coordinate system is set based on the range of the structured light camera and the physical dimensions of the structured light camera, the welding gun and the steel component.

4. The method for welding robot weld positioning based on posture estimation according to claim 1, characterized in that: The training process of the steel component pose estimation deep neural network includes: The data required for training the deep neural network for estimating the posture of steel components is constructed to train the deep neural network for estimating the posture of steel components. The deep neural network for estimating the posture of steel components obtains the posture of the steel structure by analyzing the image features of the data during the training process.

5. The method for welding robot weld positioning based on posture estimation according to claim 4, characterized in that: The data required for training the deep neural network includes a real training data set and a virtual training data set.

6. The method for welding robot weld positioning based on posture estimation according to claim 1, characterized in that: Driving the RGB camera to collect images and estimate the posture of the steel structure includes the cases where the RGB camera is fixed at the end of the robotic arm of the welding robot and fixed in the external space of the welding robot.

7. The method for welding robot weld positioning based on posture estimation according to claim 1, characterized in that: The process of extracting the weld from the three-dimensional point cloud data is specifically as follows: Down-sampling the three-dimensional point cloud data, performing plane segmentation on the down-sampled three-dimensional point cloud data and obtaining plane intersection lines; Based on the steel structure posture and the pre-acquired steel structure model, a theoretical position of a target weld is obtained; Based on the distance between the plane intersection line and the theoretical position of the target weld, the real three-dimensional coordinates of the target weld are obtained.

8. The method for welding robot weld positioning based on posture estimation according to claim 1, characterized in that: The steel component posture estimation deep neural network is a convolutional neural network or a residual neural network.

9. The method for welding robot weld positioning based on posture estimation according to claim 1, characterized in that: During the hand-eye calibration process, the calibration plate is fixed at the same position, and the welding robot is driven to make the RGB camera and the structured light camera collect multiple images containing the calibration plate in different postures, and then the hand-eye calibration process is performed.

10. A welding robot weld positioning system based on posture estimation, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor calls the computer program to execute the steps of any one of the methods according to claims 1 to 9.

Images