An automated robot repair welding system based on three-dimensional target detection and point cloud defect completion

Abstract

An automatic robot repair welding system based on three-dimensional target detection and point cloud defect completion, the whole system communicates through ROS. Through the Lidar-RCNN++ network model embedded with coordinate attention mechanism, the three-dimensional target detection of the specified workpiece is realized, combined with the PointNet network for three-dimensional defect detection, finally the PF- NET network is used for point cloud completion, and combined with the PCL point cloud processing library for defect repair welding; Ubuntu18.04 is used as the operating system, multiple deep learning algorithms are used as the basic network framework, combined with the PCL open source library for software development, and the automatic repair welding system is established. In terms of human-computer interaction, the host computer function development is combined with the QT graphical interface to establish a visual interface to display the defect workpiece point cloud, point cloud completion effect and defect repair welding result, which has the characteristics of good human-computer interaction and strong result visualization. Through this system, the problem of high-precision identification of workpiece types and workpiece defect repair welding in large-scale welding scene can be solved, and modern welding operation further moves towards automation and intelligentization.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence and industrial automation, and specifically relates to an automated robotic welding system based on three-dimensional target detection and point cloud defect completion, which can realize high-precision identification of the type of welding workpiece and welding of plate defects. Background Technology

[0002] Repair welding, as a fundamental defect plate processing technology, is widely used in actual industrial production operations and has gradually become one of the key factors affecting traditional industrial manufacturing. With the rapid upgrading of welding robots and related products such as sensors, as well as the increasing development of computer vision 3D inspection technology, improving the accuracy of 3D target detection and enhancing the quality stability of repaired workpieces has become a key research direction.

[0003] Acquiring 3D point cloud data is relatively quick, and its acquisition is unaffected by lighting conditions, avoiding the lighting and pose issues encountered with 2D images. Therefore, 3D object recognition based on point cloud data has attracted considerable attention. 3D point cloud object recognition methods typically extract geometric, shape, and structural attributes of object features, or combinations of these attributes, for comparison and learning, thereby completing object recognition and classification.

[0004] With the rapid development of welding robot technology, 3D target detection, deep learning, PCL, and other technologies, vision-based intelligent recognition and detection technologies are increasingly being applied in modern robotic welding, greatly improving the intelligence and automation of welding robots. Defect repair welding technology is one of the core technologies for achieving intelligent and automated welding. Efficient and precise welding torch repair operations are of paramount importance for ensuring the quality stability of finished workpieces. To achieve automated repair welding operations for welding robots in complex environments, the welding robot must be able to accurately identify the location of defects in workpieces under complex conditions and optimize the welding torch posture for precise repair welding. Summary of the Invention

[0005] This invention provides an automated robotic welding system based on 3D target detection and point cloud defect completion. The entire system communicates via ROS. A Lidar-RCNN++ network model with embedded coordinate attention mechanism is used to select a specific workpiece for 3D target detection from multiple workpieces. Then, a PointNet network is combined for 3D defect detection. Finally, [the system employs...].

[0006] This system utilizes the PF-NET network for defect point cloud completion and combines it with the PCL point cloud processing library for defect repair welding. It is designed for defect repair welding of workpieces in large welding scenarios. The system uses Ubuntu 18.04 as the operating system, multiple deep learning algorithms as the basic neural network framework, and the PCL point cloud processing open-source library for software development, establishing an automated repair welding system. For human-computer interaction, a host computer interaction module is designed, and the host computer functions are developed using a QT graphical interface. The resulting visual interface displays the defective workpiece point cloud, the point cloud completion effect, and the defect repair welding result, featuring good human-computer interaction and strong result visualization capabilities. This invention solves the problem of high-precision identification of workpiece types and defect repair welding in large welding scenarios, further advancing modern welding operations towards automation and intelligence.

[0007] This invention employs a modular design for the overall system, primarily comprising five modules: a data acquisition module, a 3D target detection module, a 3D defect detection module, a workpiece repair welding module, and a host computer interactive system capable of displaying the point cloud of defective workpieces, the point cloud completion effect, and the defect repair welding result. The data acquisition module uses a depth camera to acquire scene images containing multiple workpieces. It then combines the RGB and depth maps acquired by the depth camera with camera intrinsic parameters to calculate the coordinates of pixels in the camera coordinate system. By using the coordinates of obstacle points in the camera coordinate system, combined with camera intrinsic and extrinsic parameters, it obtains the point cloud data in the camera coordinate system. Simultaneously, it incorporates infrared camera checkerboard calibration, obtaining coordinate system transformation information R and T through calibration and matching. The 3D target detection module first performs preprocessing on the original scene's point cloud data, including downsampling, outlier removal, and clustering. Then, it employs a coordinate attention mechanism combined with a Lidar-RCNN++ network model using Virtualpoints to select specific workpiece point cloud data from multiple different types of workpiece point clouds. The 3D defect detection module first uses preset, defect-free workpiece labels, then classifies the model using a PointNet network model to distinguish between defective and defect-free point cloud data. The defective point cloud data is then extracted for use in the completion algorithm. The workpiece repair welding module first completes the defective point cloud using a PF-NET network, then uses a PCL library algorithm to filter out duplicate points in the point cloud data before and after completion, extracting the point cloud data of the weld points to be repaired from the defective workpiece model. Based on key point selection and orientation calculation, it accurately performs TCP welding torch repair welding operations. The host computer interactive system acquires and processes depth camera sensor data, displaying RGB images, depth maps, and point cloud data maps captured by the sensor. It can simulate the entire repair welding process and provide a correction interface for hand-eye calibration values. This patent solves the problem of defect rate detection before workpieces leave the factory, effectively reduces the labor costs required for welding operations, and greatly avoids major accidents during welding operations, improving the safety and automation of actual industrial production processes and ensuring the quality stability of finished workpieces.

[0008] The technical solution adopted by this invention to solve its technical problem is as follows:

[0009] In the data acquisition module, a depth camera is used to acquire scene images with multiple workpieces. Then, the RGB images and depth maps acquired by the RGBD camera are combined with the camera's intrinsic parameters to calculate the coordinates of each pixel in the camera coordinate system. Finally, the coordinates of obstacle points in the camera coordinate system are combined with the camera's intrinsic and extrinsic parameters to obtain point cloud data in the camera coordinate system. Simultaneously, hand-eye calibration is performed using the Zhang Zhengyou calibration method to solve for the intrinsic parameter matrix, extrinsic parameter matrix, and distortion parameters. Finally, the LM (Levenberg-Marquardt) algorithm is used to optimize these parameters, calculating the rotation matrix and translation from the camera coordinate system to the robot arm's end effector coordinate system.

[0010] In the 3D object detection module, the point cloud data in the camera coordinate system obtained from the data acquisition module undergoes point cloud preprocessing. RandomSample downsampling is used to reduce the number of points, accelerating algorithm execution. RadiusOutlierRemoval is used to remove outliers, and a region growing clustering algorithm is combined to segment the point cloud data for different types of workpieces. After point cloud preprocessing, a Lidar-RCNN++ 3D object detection network model with an improved embedded coordinate attention mechanism is used to detect specified workpiece point cloud data from multiple different types of workpiece point clouds.

[0011] In the 3D defect detection module, the system first automatically removes erroneous detection results that may exist in the 3D target detection by using preset labels. Then, the PointNet network model is used to classify the specified workpiece point cloud data through operations such as upsampling and Max Pooling, dividing the workpiece into two categories: those that need welding repair and those that do not. Finally, the point cloud data of the defective workpiece is extracted for subsequent point cloud completion.

[0012] In the workpiece repair welding module, the PF-NET point cloud completion network is first used for training and defect point cloud completion. Then, the algorithm of PCL library to remove duplicate points is used to filter out duplicate points in the point cloud data before and after completion, thereby extracting the point cloud data of the weld point to be repaired in the defective workpiece model. The starting point and ending point of the repair welding are determined by selecting key points based on the X, Y, and Z coordinates of the three-dimensional coordinates. Finally, the welding gun orientation is calculated by combining the normal vector calculation of the repair welding point cloud data and the spatial coordinate system transformation, so as to accurately perform TCP welding gun repair welding operation.

[0013] The host computer interactive system realizes the acquisition and processing of depth camera sensor data. A visual interface was designed using the Qt graphical user interface application development framework, which integrates the functions of data acquisition, workpiece recognition, point cloud completion, workpiece repair welding, and hand-eye calibration correction. Through Qt's "signal-slot" mechanism, the operation can be completed by clicking the corresponding button. It can display the RGB image, depth image, and point cloud data image captured by the depth camera, simulate the simulation image of the entire repair welding process, and provide a correction interface for hand-eye calibration values.

[0014] Compared with the prior art, the present invention, by adopting the above technical solution, has the following beneficial effects:

[0015] (1) This invention establishes a Lidar-RCNN++ network model suitable for 3D target detection by embedding a deep learning algorithm with a coordinate attention mechanism, and improves the accuracy of target workpiece detection for different workpiece models through intelligent processing.

[0016] (2) The present invention adopts a method of optimizing the welding gun orientation based on key point selection and multiple spatial coordinate system transformation, which greatly improves the accuracy and stability of the repair welding operation;

[0017] (3) This invention largely replaces the manual welding performed by technicians in the workpiece inspection scenario. At the same time, it designs an upper computer interactive system, which can facilitate the viewing of the defect welding system operation process and the modification of hand-eye calibration values. Attached Figure Description

[0018] Figure 1 This is a structural block diagram of the automated robot welding system in an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the system structure in an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of a defect repair welding model in an embodiment of the present invention.

[0021] Figure 4 This is a schematic diagram of the data acquisition module in an embodiment of the present invention.

[0022] Figure 5 This is a schematic diagram of the three-dimensional target detection module in an embodiment of the present invention.

[0023] Figure 6 This is a schematic diagram of the three-dimensional defect detection module in an embodiment of the present invention.

[0024] Figure 7 This is a schematic diagram of the workpiece repair welding module in an embodiment of the present invention.

[0025] Figure 8This is a schematic diagram of the human-computer interaction interface in an embodiment of the present invention.

[0026] Figure 9 This is a schematic diagram of the PointNet network structure in an embodiment of the present invention.

[0027] Figure 10 This is a schematic diagram of the PF-Net network structure in an embodiment of the present invention.

[0028] Figure 11 This is a schematic diagram of the Lidar RCNN++ structure in an embodiment of the present invention.

[0029] Figure 12 This is a diagram of the workpiece point cloud visualization and parameter modification interface in an embodiment of the present invention.

[0030] Figure 13 This is a diagram of the simulated actual welding test interface in an embodiment of the present invention.

[0031] Figure 14 This is a schematic diagram of the point cloud data preprocessing process in an embodiment of the present invention.

[0032] Figure 15 This is a schematic diagram of data annotation in an embodiment of the present invention.

[0033] Figure 16 This is a schematic diagram illustrating the practical operation of bar weld identification in an embodiment of the present invention.

[0034] Figure 17 This is a schematic diagram of a welding repair test in an embodiment of the present invention. Detailed Implementation

[0035] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0036] like Figure 1 and Figure 2As shown, this invention comprises five parts: a data acquisition module, a 3D target detection module, a 3D defect detection module, a workpiece repair welding module, and a human-computer interaction interface module. These modules communicate globally via a ROS publish-subscribe mechanism. The data acquisition module uses a depth camera to acquire scene images containing multiple workpieces. It then combines the RGB and depth maps acquired by the depth camera with camera intrinsic parameters to calculate the coordinates of pixels in the camera coordinate system. Finally, it uses the coordinates of obstacle points in the camera coordinate system, combined with camera intrinsic and extrinsic parameters, to obtain point cloud data in the camera coordinate system. The 3D target detection module first performs a series of preprocessing operations on the 3D point cloud data. Then, it uses a coordinate attention (CA) mechanism combined with a Lidar-RCNN++ network model using virtual points to select specific workpiece point cloud data from multiple different types of workpiece point clouds. The 3D defect detection module first presets labels for defect-free workpieces. Then, it uses a PointNet network model for model classification to distinguish between defective and defect-free point cloud data, thus obtaining defective point cloud data. The workpiece repair welding module first completes the defect point cloud using the PF-NET network. Then, it uses the PCL library algorithm to filter out duplicate points in the point cloud before and after repair, removing these duplicates to obtain the point cloud data to be repaired. Based on key point selection and welding torch orientation calculations, it accurately performs TCP welding torch repair welding operations. The human-machine interface system integrates data acquisition, workpiece recognition, point cloud completion, workpiece repair welding, and hand-eye calibration correction functions, enhancing the visibility of defective workpiece repair welding operations. The various modules collaborate to ensure the normal operation of the system.

[0037] The following details the implementation methods and functions of each module:

[0038] This system uses a combination of spot welding and bar welding for defect repair. To simplify the description of each module's function, the following methods are selected: Figure 3 The example shown is a spot welding defect repair weld, but this system is not limited to this and can be applied to all spot welding and strip welding scenarios.

[0039] Data acquisition module such as Figure 4 As shown, this module can be mainly divided into two parts: point cloud data acquisition and hand-eye calibration.

[0040] (1) The point cloud data acquisition part first uses an RGBD camera to acquire scene images with multiple workpieces. The acquired RGB image and depth map are combined with the camera intrinsic parameters. Based on the X and Y coordinates (u and v in the formula) of the pixel coordinate system provided by the RGB image and the camera intrinsic parameters, the X and Y coordinate values ​​in the camera coordinate system are calculated. At the same time, the depth map directly provides the Z coordinate value in the camera coordinate system. Combining the two, the coordinates P = [XYZ] in the camera coordinate system are obtained. T The coordinates of the obstacle points in the camera coordinate system are the point cloud sensor data, which is the point cloud data in the camera coordinate system.

[0041] Camera coordinate system P and pixel coordinate system P uv The formula relating the coordinates of points below, where f is... x f is the reciprocal of the physical size of each pixel in the x-direction of the image plane. y The reciprocal of the physical size of each pixel in the y-direction of the image plane, c x c y These are the coordinates of the origin of the image coordinate system in the pixel coordinate system:

[0042]

[0043] The specific solution formula after rearranging the above equation is as follows:

[0044]

[0045] Z = d (4)

[0046] Generally speaking, the Z value of the point in the formula in the camera coordinate system is the depth value d measured by the camera, which is the distance from the real point to the camera plane. If it is not, it can be compensated by multiplying the value. (2) The hand-eye calibration part first collects 200 images of the calibration board with a 22mm grid size. The feature points in the images, such as the corner points of the calibration board, are detected to obtain the pixel coordinate values ​​of the corner points of the calibration board. Based on the known size of the checkerboard and the origin of the world coordinate system, the physical coordinate values ​​of the corner points of the calibration board are calculated. Based on the relationship between the physical coordinate values ​​and the pixel coordinate values, the H matrix is ​​obtained, and then the v matrix is ​​constructed. The B matrix is ​​solved, and the camera intrinsic parameter matrix A is solved using the B matrix. Finally, the camera extrinsic parameter matrix corresponding to each image is solved, thereby obtaining R and T (R is the rotation matrix and T is the translation amount). At the same time, the matrix can be constructed to calculate the radial distortion parameters. After the calibration is completed, the intrinsic parameter matrix, translation and rotation parameters and distortion parameters will be obtained. Finally, the LM (Levenberg-Marquardt) algorithm can be used to optimize the above parameters and calculate the rotation matrix and translation amount from the camera coordinate system to the end effector coordinate system of the robotic arm.

[0047] 3D target detection module, such as Figure 5 As shown, this module can be mainly divided into two parts: point cloud preprocessing and 3D target detection.

[0048] (1) Point Cloud Preprocessing. This part utilizes the PCL open-source point cloud processing library to perform three preprocessing operations on the point cloud data: downsampling, outlier removal, and clustering segmentation. Since the large amount of point cloud data acquired by the sensor affects the algorithm's running speed, a random downsampling method is used to reduce the number of points. 80,000 points are selected from the original point cloud data, with each point having an equal probability of being selected, resulting in 80,000 point clouds with the specified number of sampling points. Because the randomly downsampled point cloud data may filter out some key feature points with important information, a fixed random seed of 20 is used to generate a fixed random number sequence to ensure the scientific validity of the random downsampling and improve the execution speed of subsequent algorithms without losing important point cloud features. Outlier removal uses a RadiusOutlierRemoval filter for each point P in the point cloud. i Determine a neighborhood of radius r (i.e., with respect to P) i (where r is the center of a sphere), if the number of points in its neighborhood is N... <N threshold Then we consider point P to be... i Noise points are identified and removed. The clustering segmentation uses a region-growing-based algorithm. First, an empty seed point sequence and an empty clustering array are set. A seed point (the point with the smallest curvature) is selected and added to the seed point sequence. Neighboring points of the current seed point are searched, and the angle between the normal of a neighboring point and the normal of the current seed point is calculated. Neighboring points with a curvature less than a smoothing threshold are added to the current region. The curvature of each neighboring point is checked; those with a curvature less than a curvature threshold are added to the seed point sequence, and the current seed point is deleted. Growth continues with a new seed point. This growth process is repeated until the seed point sequence is cleared. At this point, a region has been grown and added to the clustering array. The above steps are repeated for the remaining points until all points have been traversed. Using this PCL-based point cloud preprocessing algorithm, individual point cloud data for each workpiece can be separated.

[0049] (2) 3D Object Detection. This part first builds a Lidar-RCNN++ network model that embeds a coordinate attention (CA) mechanism and uses virtual points, such as... Figure 5As shown, Lidar-RCNN++ can be divided into three stages: PointNet, Lidar-RCNN, and Lidar-RCNN++. PointNet is the first deep model framework to directly process unordered point cloud data. The input is 3D point cloud data containing n points. A 3D transformation matrix prediction network, T-Net, estimates a 3*3 transformation matrix T and applies it to the original data, achieving data alignment. The aligned data is then processed point-by-point using a shared parameter two-layer perceptron model, Shared MLP, for feature extraction. Each point extracts 64-dimensional features, which are then passed through T-Net and Shared MLP until the feature dimension becomes 1024 dimensions. MaxPooling is then performed on the feature space to extract the global feature vector of the point cloud. Lidar-RCNN uses the PointNet network as its backbone network and employs the method of adding virtual points to solve the problem that the blank parts in the PointNet proposal cannot be represented in the features, only extracting and fusing features from the scanned points, and having no awareness of the proposal size. Lidar-RCNN++ embeds a coordinate attention mechanism on top of the Lidar-RCNN network model, which can capture global contextual dependencies and significantly improve the accuracy of 3D object detection. The coordinate attention mechanism is an efficient way to capture positional information and channel relationships to enhance the feature representation of the Mobile Network. By decomposing the 3D global pooling operation into three one-dimensional encoding processes, CA outperforms other attention methods with lightweight properties (such as SENet, CBAM, and TA).

[0050] CoordinateAttention encodes channel relationships and long-term dependencies using precise location information. The specific operation consists of two steps: Coordinate information embedding and CoordinateAttention generation.

[0051] ① Global pooling is typically used for global encoding of spatial information in channel attention encoding. However, because it compresses global spatial information into the channel descriptor, it makes it difficult to preserve positional information. To enable the attention module to capture long-range spatial interactions with precise positional information, CA decomposes global pooling into a one-to-one one-dimensional feature encoding operation using the following formula:

[0052]

[0053] Specifically, given an input X, each channel is first encoded along the horizontal and vertical coordinates using a pooling kernel of size (H,1) or (1,W). Therefore, the output of the c-th channel at height h can be expressed as:

[0054]

[0055] Similarly, the output of the c-th channel with width w can be written as:

[0056]

[0057] The two transformations described above aggregate features along two spatial directions, resulting in a pair of direction-aware feature maps. This is quite different from the SE Block, which generates a single feature vector in channel attention methods. These two transformations also allow the attention module to capture long-term dependencies along one spatial direction and preserve precise location information along the other, which helps the network to more accurately locate the target of interest.

[0058] ②CoordinateAttention generates the transformations from the information embedding. This part concatenates these transformations and then applies a convolutional transformation function, where δ is a non-linear activation function, F1 is a 1×1 convolutional transformation function, and z... h It is the output of the c-th channel with height h, z w It is the output of the c-th channel with width w, σ is the sigmoid function, and f h f w Let f be split into two independent tensors along the spatial dimension, F w F h It is a 1×1 convolution transformation that will transform f h f w Convert to a tensor with the same number of channels as the input X:

[0059] f=δ(F1([z h ,z w ])) (8)

[0060] g h =σ(F h (f h (9)

[0061] g w =σ(F w (f w (10)

[0062] Finally, the output Y of the Coordinate Attention Block can be written as:

[0063]

[0064] 3D defect detection module, such as Figure 6 As shown, firstly, the system automatically removes possible erroneous detection results in 3D target detection using preset labels. Then, it uses the PointNet network model to perform model classification on the specified workpiece point cloud data through operations such as upsampling and Max Pooling, dividing the workpieces into two categories: those that need welding repair and those that do not. Finally, it extracts the point cloud data of the defective workpieces for subsequent point cloud completion.

[0065] Workpiece repair welding module, such as Figure 7 As shown, the PF-NET point cloud completion network is first used to complete the defective point cloud. The general idea is to take the defective point cloud data as input and output the completed point cloud generated based on the input. However, the generated point cloud is inherently uncertain whether it is correct. Therefore, these generated point clouds are then put into a discriminator to ultimately determine whether the completion result is True or False.

[0066] The PF-NET network model framework can be divided into two parts: a generator and a discriminator.

[0067] ① The generator part first uses the farthest point sampling point cloud extraction method to implement FPN (multi-scale feature fusion). Then, PF-Net performs a MaxPooling operation on the result of each feature dimensionality upscaling in the point cloud feature extraction part. After that, the Max Pooling results of each layer are concatenated to become the feature extraction result. Finally, the feature extraction results of the same point cloud but different densities are concatenated again, followed by an MLP operation and a convolution operation to obtain the final feature extraction result. After obtaining the feature extraction result, the point cloud completion generation stage begins. First, the obtained feature extraction result is passed through two fully connected layers one after another. The deepest FC3 is used to generate the skeleton of the point cloud. Then, FC3 and FC2 are added together to generate the flesh of the point cloud. Finally, the three feature extraction results FC3, FC2 and FC1 are added together to complete the complete point cloud data.

[0068] ② After obtaining the complete and dense point cloud completion result, the discriminator compares this result with the actual ground truth point cloud result, ultimately returning a True or False boolean value to determine the accuracy of the generated completed point cloud. In the PF-Net network model diagram, this comparison is abbreviated as CD, and its formula is shown below, where S1 represents the predicted point cloud, S2 represents the ground truth point cloud, and d... CD Represents the average squared nearest distance, where x and y represent coordinate values:

[0069]

[0070] Since the PF-Net network is essentially a GAN (Generative Adversarial Network), it also follows the loss function of GANs, where D() is the output of the discriminator, yi represents the ground truth, and yi* is the predicted value.

[0071]

[0072] After obtaining the correct completed point cloud model through PF-Net, this invention combines the algorithm of PCL library to remove duplicate points to filter out duplicate points in the point cloud data before and after completion, thereby extracting the point cloud data of the weld points to be repaired in the defective workpiece model. The core principle of the algorithm is to use a balanced binary tree KD-Tree to determine if a point is not the same as itself in a threshold neighborhood at a certain distance, then the two are duplicate points.

[0073] To ensure the verticality of the welding torch during actual welding operations, normal vector estimation was performed on the point cloud data of the points to be repaired. OpenMP was used to accelerate normal estimation, with 10 threads. A lightweight binary tree, KD-Tree, was used for fast nearest neighbor search. The average distance between any point in the point cloud and its 70 nearest neighbors was set to follow a Gaussian distribution. Finally, the normal vector of the repaired area was calculated. After obtaining the normal vector, it was found that the normal vectors of some points to be repaired were inaccurate. An averaging method was used to further refine the direction of the normal vector. The rotation matrix R and translation T from the camera coordinate system to the Cam2gripper coordinate system of the robotic arm's end effector were obtained through hand-eye calibration. The average normal vector in the camera coordinate system was then transformed to the robotic arm's end effector coordinate system through rotation and translation. A new custom coordinate system is constructed using the reverse of the average normal vector of the robotic arm's end-effector coordinate system, normalized to Vz. The vector representing the difference between the starting and ending points of the welding area in the end-effector coordinate system is normalized to Vx. A keypoint selection process is performed to re-determine the starting and ending points of the welding area. The cross product of Vz and Vx is then used as Vy. The custom coordinate system under the robotic arm's end-effector coordinate system is thus completed. To verify whether the three axes are pairwise perpendicular, the `getAngle3D` function is introduced to calculate the angular relationship between two vectors. To calculate the rotation matrix R and translation T from the custom coordinate system to the robotic arm's end-effector coordinate system, the following formula is used, where Vx, Vy, and Vz are the normalized vectors of each axis, and `transpose` is the transpose of the vectors.

[0074] R = (V x V y V z ).transpose() (14)

[0075] With Vx V y V z The transpose of the vector is used as the rotation matrix R, and MinPoint_cam2gripper is used as the translation amount T. To obtain the rotation matrix R in the base coordinate system, the rotation matrix R_gripper2base from the end effector of the robot arm to the base coordinate system needs to be multiplied by the above rotation matrix to obtain the rotation matrix from the custom coordinate system to the base coordinate system. Finally, the rotation matrix is ​​converted to Euler angles using the eulerAngles function. The topic of the start and end points of the welding area and the Euler angles is published through the ROS node. The host computer subscribes to the topic, reads the start and end points of the welding area and the Euler angles information, and returns it to the robot arm for actual TCP welding operations.

[0076] Human-computer interaction interface modules such as Figure 8 As shown, the acquisition, processing, and transmission of sensor data are realized. A visual interface was designed using the QT graphical user interface application development framework, integrating functions such as data acquisition, workpiece recognition, point cloud completion, workpiece welding repair, and hand-eye calibration correction. Through QT's "signal-slot" mechanism, operations are completed by clicking the corresponding buttons. When the user clicks the data acquisition button, the RGB image, depth map, and point cloud data image captured by the depth camera are displayed; when the user clicks the workpiece recognition button, the point cloud PCD image of the detected target workpiece is displayed; when the user clicks the point cloud completion button, the workpiece point cloud data after completion is displayed; when the user clicks the workpiece welding repair button, a simulation diagram of the entire welding process is displayed; when the user clicks the hand-eye calibration correction button, a correction interface for hand-eye calibration coordinates (X, Y, Z) and Euler angles (W, P, R) is provided.

[0077] The advantages of the system proposed in this invention are as follows:

[0078] (1) A feasible solution was proposed for an intelligent defect repair welding system in modern industrial production, which greatly improved the quality stability of products before they leave the factory.

[0079] (2) By performing repair welding operations on defective workpiece data collected and detected in real time, labor costs are effectively reduced, while the safety of construction workers at the grassroots level is improved.

[0080] (3) This system has a host computer interaction function, which improves the visibility of the welding repair operation and adds intuitive, easy-to-operate and easy-to-modify calibration function, which can more efficiently and intuitively cooperate with technicians to supervise and operate the welding repair operation.

[0081] This invention discloses an automated robotic welding system based on 3D target detection and point cloud defect completion. It solves the problem of accurately identifying workpieces and their defects and achieving automated welding in large-scale defect welding scenarios, further advancing modern welding operations towards intelligence. Simultaneously, it can reduce labor costs for enterprises in the welding field and significantly reduce the occurrence of workshop safety accidents, greatly improving the safety and automation level of industrial production.

[0082] The advantages of this invention lie in its use of a deep learning algorithm based on a coordinate attention mechanism to establish a Lidar-RCNN++ network model suitable for 3D object detection, and in its use of an algorithm based on key point selection and multiple spatial coordinate system transformations to optimize the welding torch orientation. This solves the problem of ensuring the stability of finished workpiece quality in workpiece repair welding scenarios, further advancing the intelligentization of defective workpiece repair welding operations. A host computer interaction system is designed to enhance the visibility of defective workpiece repair welding operations and provides a more convenient hand-eye calibration and correction method. This system integrates artificial intelligence and computer vision technology into the actual factory production environment, representing a real-world example of modern AI technology being applied to practical industry, and ensuring a crucial production safety link in actual industrial production processes.

[0083] The deployment and experimentation of this system were conducted using a Lenovo Legion R9000P as the computing platform, equipped with a 3070 series graphics card. Development was carried out on the Ubuntu operating system, with the graphics card driver, CUDA 10.1, CUDNN, OpenCV, PCL, Eigen, etc., installed. PyTorch was used as the framework for dataset annotation and training, and IDEs such as Clion and PyCharm were used for debugging and training. Figure 12 A diagram showing the interface for visualizing the workpiece point cloud and modifying parameters. Figure 13 To simulate the actual welding test interface, the starting and ending point coordinates and the welding torch perpendicularity are determined in this interface. Figure 14 This is a schematic diagram of the point cloud data preprocessing process, which involves radius filtering and region growing algorithms for processing the point cloud data. For details, please refer to the point cloud preprocessing section above. Figure 15 This is a diagram illustrating data labeling, due to network training. Figure 16 This is a practical diagram of bar weld identification, where the vertical lines mark the weld repair locations, which are accurately identified by the trained network model. Figure 17 This is a schematic diagram of a practical welding test, showing how the actual welding location is processed using a welding torch.

[0084] The above description is only a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. Any equivalent modifications or changes made by those skilled in the art based on the content disclosed in the present invention should be included within the scope of protection set forth in the claims.

Claims

1. An automated robot repair welding system based on three-dimensional target detection and point cloud defect completion, characterized in that: the system comprises a data acquisition module, a three-dimensional target detection module, a three-dimensional defect detection module, a workpiece repair welding module and a man-machine interaction interface module; the data acquisition module acquires workpiece images using a depth camera, and then obtains point cloud data in the camera coordinate system by combining the RGB image, the depth image and the camera intrinsic parameters acquired by the depth camera; the three-dimensional target detection module first performs preprocessing operation on the three-dimensional point cloud data, and then uses the Lidar-RCNN++ network model based on the coordinate attention mechanism and combined with Virtual points to select the specified workpiece point cloud data from multiple different types of workpiece point clouds; the three-dimensional defect detection module first presets a defect-free workpiece label, and then uses the PointNet network model for model classification to distinguish the defective point cloud data from the defect-free point cloud data and obtain the defect point cloud data; the workpiece repair welding module first performs defect point cloud completion through the PF-NET network, then screens out the repeated points before and after the completion of the point cloud by combining the PCL library algorithm, removes the repeated points to obtain the point cloud data to be repaired, and performs TCP welding gun repair welding operation according to the key point selection and welding gun orientation calculation; the PF-NET network model framework is divided into two parts of generator and discriminator, the defective point cloud data is taken as input, and the completed point cloud is generated according to the input; the generator part first uses the farthest point sampling point cloud extraction method to realize the scale feature fusion FPN, then the PF-Net performs a Max Pooling operation on the result of each feature dimension increase in the point cloud feature extraction part, then the Max Pooling results of each layer are spliced to become the feature extraction result, and finally the same point cloud and different density feature extraction results are spliced, and then an MLP operation and a convolution operation are performed to obtain the final feature extraction result; after obtaining the feature extraction result, the generation stage of the completed point cloud is entered, the obtained feature extraction result FC1 is sequentially passed through two fully connected layers to obtain FC2 and FC3, wherein the feature extraction result FC3 is used to generate the skeleton of the completed point cloud, then the feature extraction results FC3 and FC2 are added to generate the flesh of the completed point cloud, and then the three feature extraction results FC3, FC2 and FC1 are added to complete the point cloud data; The discriminator part compares the complete and dense point cloud completion result with the actual real point cloud result after obtaining the complete and dense point cloud completion result, and finally returns a Boolean value of True or False to determine the accuracy of the generated completed point cloud. The formula is as follows, wherein represents the predicted point cloud, represents the true value point cloud, represents the average nearest square distance, , represents the coordinate value: the PF-Net follows the loss function of the generative adversarial network GAN; the man-machine interaction system integrates and realizes the functions of visual data acquisition, workpiece recognition, point cloud completion, workpiece repair welding and hand-eye calibration correction.

2. The automated robotic repair welding system based on three-dimensional target detection and point cloud defect completion of claim 1, wherein: The data acquisition module is a multi-sensor fusion module, which uses an RGBD depth camera combined with an infrared camera for data acquisition and hand-eye calibration. In the data acquisition, the RGB image and the depth image are acquired by the RGBD depth camera, and the point cloud data in the camera coordinate system is obtained by combining the camera coordinate system and the pixel coordinate system. In the hand-eye calibration, the feature points in the image are detected to obtain their pixel coordinate values, the corresponding physical coordinate values are calculated, the camera extrinsic matrix corresponding to each picture is solved, the intrinsic matrix, translation and rotation parameters and distortion parameters are obtained through calibration, and the rotation matrix and translation of the camera coordinate system to the welding torch mechanical arm end coordinate system are calculated.

3. The automated robotic repair welding system based on three-dimensional target detection and point cloud defect completion of claim 1, wherein: The three-dimensional target detection module is divided into a point cloud preprocessing part and a three-dimensional target detection part; The point cloud preprocessing part uses the PCL point cloud processing open source library to perform point cloud data downsampling, outlier removal and clustering segmentation on the point cloud data; The three-dimensional target detection part builds a Lidar-RCNN++ network model embedded with a coordinate attention mechanism to detect and classify different types of plates with high precision. The Lidar-RCNN++ network model is composed of a three-dimensional space transformation matrix prediction network T-Net, a shared parameter double-layer perception machine model shared mlp, virtual points, MaxPooling and a deformation of the coordinate attention mechanism CA.

4. The automated robotic repair welding system based on three-dimensional target detection and point cloud defect completion of claim 1, wherein: The three-dimensional defect detection module is based on the Pointnet classification and regression network of pure point cloud, classifies the workpiece point cloud data, divides the workpiece into two categories of needing repair welding and not needing repair welding, and finally extracts the defective workpiece point cloud data for subsequent repair welding.

5. The automated robotic repair welding system based on three-dimensional target detection and point cloud defect completion of claim 1, wherein: The workpiece repair welding module is divided into a point cloud completion part, a repeated point cloud removal part and an intelligent repair welding part. The point cloud completion part uses the PF-NET network to divide the point cloud completion into generation and discrimination, and after completing the point cloud, the completed data is discriminated again to increase the confidence of the completed area. The repeated point cloud removal part combines the KD-Tree threshold neighborhood repeated point removal algorithm in the PCL point cloud processing algorithm library to effectively extract the point cloud data to be repaired. The intelligent repair welding part uses the OpenMP parallel computing method to calculate the normal vector of the point cloud data to be repaired, extracts the key points of the repair area, and performs multiple three-dimensional space coordinate system transformations on the normal vector.

6. The automated robotic repair welding system based on three-dimensional target detection and point cloud defect completion of claim 1, wherein: The human-computer interaction interface module is a visual interface developed based on Qt, which integrates the functions of data acquisition, workpiece recognition, point cloud completion, workpiece repair welding and hand-eye calibration correction.

7. The automated robotic repair welding system based on three-dimensional target detection and point cloud defect completion of claim 6, wherein: The data acquisition button displays the RGB image, depth image and point cloud data image captured by the depth camera in real time.

8. The automated robotic repair welding system based on three-dimensional target detection and point cloud defect completion of claim 6, wherein: The workpiece repair welding button simulates the whole repair welding process to realize the supervision and operation of the repair welding operation.

Images