A deep learning-based intelligent seam guiding welding system and method

Abstract

This invention proposes a deep learning-based intelligent guided welding system and method. The method includes: constructing a welding system composed of a six-axis industrial robot, a vision perception module, an industrial control computer, and a robot controller; performing system initialization and calibration; acquiring weld seam and laser stripe images, and dividing them into training and validation sets; constructing a weld seam feature extraction model based on an improved YOLOv8, and training the model using the training set; acquiring weld seam and laser stripe images from the actual welding site, and using the trained improved YOLOv8 weld spot detection model for detection, outputting the weld spot's bounding box position, confidence level, classification result, and accurate two-dimensional pixel coordinates of key points; converting the two-dimensional pixel coordinates of key points to three-dimensional coordinates in the robot's base coordinate system; and transmitting the actual three-dimensional coordinates of the weld spot and the welding torch posture to the robot controller as path points to plan and execute the welding trajectory, completing the automated welding operation.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent welding, and relates to a welding seam intelligent guiding welding system and method based on deep learning. BACKGROUND

[0002] With the rapid development of industrial automation and intelligent manufacturing, higher requirements are put forward for the precision, efficiency and adaptability of welding processing, and it has become a key development direction to use visual technology to guide a welding robot to realize intelligent operation.

[0003] In the prior art of robot intelligent welding, one of the representative schemes is the method disclosed in Chinese patent "CN110480127B". The specific implementation process of the scheme is as follows: a line structured light laser is used to project a laser stripe onto a welding seam, and an image containing the stripe is collected by a camera; then, a series of preset traditional image processing algorithms are used to process the laser stripe, and finally the two-dimensional coordinates of the feature points of the welding seam are calculated. The visual recognition method based on traditional image processing represented by the patent has obvious defects. The scheme relies heavily on preset image processing algorithms and clear laser stripe features. In actual industrial environments, when encountering strong arc light interference, smoke diffusion, workpiece surface reflection or complex and variable welding seam types, the quality of the laser stripe image will decrease significantly, resulting in failure of feature extraction or sharp reduction of positioning accuracy of the traditional algorithm. In addition, the method lacks self-adaptive learning ability, and needs to adjust the algorithm parameters every time a new welding seam type or working condition changes, which has poor flexibility and is difficult to meet the requirements of modern intelligent manufacturing for high adaptability and high robustness of welding robots. SUMMARY

[0004] To solve the above technical problems, the purpose of the present application is to provide a welding seam intelligent guiding welding system and method based on deep learning, which realizes stable and accurate identification and guidance of the positions of various types of welding points by using the powerful feature learning ability of deep learning, and improves the intelligent level and practical reliability of the welding robot system in real industrial environments.

[0005] The present application provides a welding seam intelligent guiding welding system based on deep learning, comprising a six-axis industrial robot, a visual perception module, an industrial computer, a robot controller, a welding gun and a welding execution module.

[0006] The six-axis industrial robot is used to carry the welding gun and the visual perception module for precise movement, the welding gun is installed at the end of the robot, and the visual perception module is fixed on the welding gun tube.

[0007] The visual perception module comprises a frame and an industrial camera, a line laser and a light filtering unit mounted on the frame, and is used to collect welding seam and laser stripe images.

[0008] The industrial computer is used for pre-processing weld seam and laser stripe images, and extracting image features to obtain two-dimensional pixel coordinates of the weld seam.

[0009] The robot controller is responsible for receiving the two-dimensional pixel coordinates from the industrial computer, converting to three-dimensional coordinates in the robot base coordinate system, planning and motion control of the six-axis industrial robot, and feeding back the pose state of the robot to the industrial computer in real time.

[0010] The welding execution module includes a welding power supply and a protective gas bottle, and is responsible for completing the cladding welding process.

[0011] The application also provides a welding seam intelligent guiding welding method based on deep learning, comprising:

[0012] Step 1: build a welding system composed of a six-axis industrial robot, a visual perception module, an industrial computer, a robot controller, a welding gun and a welding execution module;

[0013] Step 2: system initialization and calibration;

[0014] Step 3: collect weld seam and laser stripe images through the visual perception module, and pre-process them, and divide them into training set and validation set according to the ratio of 8:2;

[0015] Step 4: build a weld seam feature extraction model based on improved YOLOv8, and train the weld seam feature extraction model based on improved YOLOv8 using training set data;

[0016] Step 5: evaluate the performance of the weld seam feature extraction model based on improved YOLOv8 based on the validation set;

[0017] Step 6: collect weld seam and laser stripe images in actual welding site, detect them using the trained weld point detection model based on improved YOLOv8, and output the boundary box position, confidence, classification result and accurate key point two-dimensional pixel coordinates of the weld point;

[0018] Step 7: convert the key point two-dimensional pixel coordinates to three-dimensional coordinates in the robot base coordinate system, i.e. the actual three-dimensional coordinates of the weld point position;

[0019] Step 8: transmit the actual three-dimensional coordinates of the weld point position and the welding gun pose to the robot controller in real time, take them as path points, plan and execute the welding trajectory, and complete the automatic welding work.

[0020] The welding seam intelligent guiding welding system and method based on deep learning has the following beneficial effects:

[0021] (1) Innovative design of deep learning model for complex welding scenarios: By introducing advanced structures such as deformable convolution, dual-path attention mechanism, and content-aware upsampling, an improved YOLOv8 model specifically for weld feature extraction was constructed. This model has the ability to dynamically adapt to feature deformation, autonomously focus on key areas, and preserve detailed information, fundamentally improving the robustness and accuracy of feature extraction under interference conditions such as strong arc light, smoke, and spatter.

[0022] (2) Intelligent weld recognition mechanism based on multi-task learning: A multi-task learning head network was designed to simultaneously complete weld location, feature point extraction, and weld type classification. Through feature sharing and knowledge transfer, this mechanism enables the model to accurately identify various weld types that may occur in the same welding process, such as V-groove, fillet weld, and butt weld I-groove, and outputs feature point coordinates with sub-pixel accuracy, thus achieving a complete understanding and description of complex welding paths.

[0023] (3) A complete and high-precision vision guidance chain: from precise camera, hand-eye and light plane calibration, to innovative feature extraction, and then to coordinate transformation including process compensation, a reliable and accurate solution is formed.

[0024] (4) Engineering-based system integration and interaction: It provides practical functions such as point scanning planning, special point processing, and manual coordinate correction, enabling the system to flexibly cope with complex situations such as inconsistent workpiece assembly in actual welding. Attached Figure Description

[0025] Figure 1 This is a flowchart of a deep learning-based intelligent guided welding method for weld seams according to the present invention.

[0026] Figure 2 This is a framework diagram of the weld feature extraction model based on the improved YOLOv8 of the present invention;

[0027] Figure 3 This is a structural diagram of the C2f_DCNv3 module;

[0028] Figure 4 This is a structural diagram of a lightweight dual-path attention module;

[0029] Figure 5 This is a schematic diagram of the principle of linear laser measurement.

[0030] Figure 6 This is a flowchart of the multi-task header and KPLoss calculation process. Detailed Implementation

[0031] This invention discloses a deep learning-based intelligent guided welding system for weld seams, whose hardware architecture forms the foundation for the welding method. The system uses a six-axis industrial robot as its core motion control unit, integrating visual perception, welding execution, and control units. Through precise calibration and communication protocols, it forms a closed-loop automated welding workstation.

[0032] The system mainly includes the following modules: a six-axis industrial robot, a vision perception module, an industrial computer, a robot controller, a welding torch, and a welding execution module.

[0033] The six-axis industrial robot is used to carry a welding torch and a vision perception module for precise movement. The welding torch is mounted at the robot's end effector, and the vision perception module is fixed to the torch tube. The vision perception module includes a frame and an industrial camera, a line laser, and a filter unit mounted on the frame, responsible for acquiring images of the weld seam and laser stripes. The industrial control computer is used to preprocess the weld seam and laser stripe images and extract image features to obtain the two-dimensional pixel coordinates of the weld seam. The industrial control computer establishes TCP / IP communication with the robot controller via Ethernet, and implements the functions of sending coordinate commands and receiving status information by writing corresponding Socket communication programs. The robot controller is responsible for receiving the two-dimensional pixel coordinates from the industrial control computer, converting them to three-dimensional coordinates in the robot's base coordinate system, performing trajectory planning and motion control of the six-axis industrial robot, and providing real-time feedback on the robot's pose status to the industrial control computer. The welding execution module includes a welding power supply and a protective gas cylinder, responsible for completing the cladding welding process.

[0034] In specific implementation, the six-axis industrial robot is a KR2010 six-axis arc welding robot with a repeatability accuracy of ±0.08mm, sufficient to meet the requirements of high-precision welding. The industrial camera is a MER2-137-90GM-P industrial digital camera equipped with an M1214-MP2 fixed-focus lens to ensure image clarity and stable field of view. The laser used is a 150mW 650nm single-line red laser, whose core purpose is to project structured light stripes onto the weld, thereby converting the depth, contour, and other three-dimensional geometric information of the weld into clear two-dimensional visual features. For different types of welds (such as V-type, butt, and filler layer welds), the laser stripes undergo unique deformation due to changes in the workpiece surface shape, resulting in light stripe images with distinctly different shape characteristics. This provides crucial information for subsequent image processing and weld type identification. The filter unit includes a 650nm red light lens and a polarizing lens, which are sequentially mounted on the outside of the lens to effectively suppress environmental interference such as welding arc light and fumes, highlighting the laser stripes. The D500S gas shielded welding power source is selected. The shielding gas required for welding is a mixture of 80% Ar + 20% CO2.

[0035] likeFigure 1 As shown, the present invention provides a deep learning-based intelligent guided welding method for weld seams, comprising:

[0036] Step 1: Build a welding system consisting of a six-axis industrial robot, a vision perception module, an industrial computer, a robot controller, a welding torch, and a welding execution module.

[0037] Step 2: Perform system initialization and calibration. This step is a prerequisite for system operation and aims to establish a spatial mapping relationship from the pixel coordinates of the two-dimensional image to the robot's three-dimensional base coordinate system. Specifically:

[0038] Step 2.1: Camera calibration. A checkerboard calibration board of specific specifications is used, for example, a 12×9 checkerboard calibration board with each square having an actual size of 5mm×5mm. By changing the pose of the checkerboard calibration board relative to the visual perception module, images of the checkerboard calibration board in multiple poses are acquired to ensure sufficient pose diversity. Then, the images are imported into the MATLAB camera calibration toolbox. The pixel coordinates of all checkerboard corner points are obtained through an automatic corner extraction algorithm, and the world coordinates corresponding to the corner points are determined based on the known physical dimensions of the checkerboard. Based on these corresponding pixel coordinates and 3D world coordinates, the Zhang Zhengyou calibration method is used to preliminarily calculate the camera's intrinsic parameter matrix (including focal length and principal point coordinates), extrinsic parameter matrix (i.e., the rotation and translation vectors of the calibration board relative to the camera in each image), and lens distortion coefficients, thus achieving camera calibration.

[0039] Step 2.2: Hand-eye calibration. Determine the relative pose between the vision perception module and the robot's end effector welding torch. By controlling the welding torch, move the vision perception module to multiple different poses, capture images of the calibration board fixed in the workspace, and simultaneously record the transformation matrix of the robot's end effector coordinate system relative to the base coordinate system. W T E The transformation matrix of the calibration board coordinate system relative to the camera coordinate system in each image is extracted using a camera calibration algorithm. C T B Based on the coordinate system transformation relationship, the hand-eye equation is established, and the Tsai two-step method is used to solve the equation. The rotation matrix and translation vector are calculated sequentially to completely determine the rigid transformation relationship from the camera coordinate system to the robot end effector coordinate system. E T C Complete hand-eye calibration.

[0040] Step 2.3: Laser plane calibration. Maintaining the fixed pose of the visual perception module, multiple sets of chessboard images projected with and without laser stripes are simultaneously acquired by alternately switching the line laser. The calibration board image without laser stripes is imported into the MATLAB camera calibration toolbox to obtain the camera extrinsic parameters corresponding to each set of images. Simultaneously, the Steger algorithm is used to extract the center line of the laser stripes in the images with laser stripes, and their pixel coordinates are converted to the image coordinate system to calculate the linear equation of the laser stripes in the image. Combining the obtained camera extrinsic parameters, the pixels on the center line of the laser stripes are converted to the camera coordinate system. The least squares method is used to perform plane fitting on these three-dimensional coordinate points to obtain the linear polynomial equation of the laser plane in the camera coordinate system, thus completing the high-precision calibration of the laser plane.

[0041] Step 3: Acquire images of the weld seam and laser stripes using the visual perception module, and preprocess them. Divide the images into training and validation sets in an 8:2 ratio, specifically:

[0042] Step 3.1: Control the six-axis industrial robot to move to the predefined scanning point so that the feature area of ​​the weld seam image is displayed on the image screen.

[0043] Step 3.2: Project structured light stripes onto the weld to convert the three-dimensional geometric information of the weld into clear two-dimensional visual features.

[0044] Step 3.3: While acquiring images using an industrial camera, continuously change the robotic arm angle, image display exposure time, contrast, and gain effect to acquire multiple images of various weld types, such as V-type, corner joint, butt joint I-type, V-type with gap, and filler layer welds and laser stripe images.

[0045] In practice, the types of images collected include almost interference-free weld seam images collected with filters, images with background interference collected without filters, strong interference images with arc welding light and spatter during welding, and weld seam images under constantly changing exposure time and gain effects. A total of 2,500 weld seam images are collected for each type of weld seam, and 500 weld seam images are collected for each of the above five types of collection environments.

[0046] Step 3.4: Use Labelme software to label the weld point positions of the collected weld seam and laser stripe image data, and divide them into training set and validation set at an 8:2 ratio.

[0047] Step 4: Construct a weld feature extraction model based on the improved YOLOv8, and train the improved YOLOv8 weld feature extraction model using the training set data.

[0048] In actual welding processes, strong arc light, fumes, spatter interference, and complex and varied weld seam types are frequently encountered, causing traditional laser stripe centerline extraction algorithms (such as the Steger method) to fail in low-quality image areas. To address this issue, this invention constructs a weld seam feature extraction model based on an improved YOLOv8 and performs a series of deep optimizations for industrial welding scenarios, enabling it to robustly extract sub-pixel-level weld seam feature point coordinates directly from noisy laser stripe images.

[0049] like Figure 2 As shown, the weld feature extraction model based on the improved YOLOv8 includes a backbone network, a neck network, and a head network. The backbone network includes two basic convolutional modules, four C2f_DCNv3 modules, three downsampling convolutional modules based on convolution operations, and a lightweight dual-path attention module. The original weld image data is sequentially input into the first convolutional module, the second convolutional module, the first C2f_DCNv3 module, the first downsampling convolutional module, the second C2f_DCNv3 module, the second downsampling convolutional module, the third C2f_DCNv3 module, the third downsampling convolutional module, the fourth C2f_DCNv3 module, and the lightweight dual-path attention module of the backbone network for feature extraction.

[0050] In specific implementation, YOLOv8 was improved based on the improved weld feature extraction model as follows:

[0051] 1) A deformable convolution mechanism was introduced into the backbone and neck networks of the YOLOv8 model, and the original C2f module was optimized into the C2f_DCNv3 module.

[0052] like Figure 3 As shown, in the C2f_DCNv3 module, the input feature map is convolved and then copied into two identical parts. The first part of the feature map is input into multiple sequentially connected Bottleneck_DCNv3 modules containing deformable convolutions for processing. Each Bottleneck_DCNv3 module incorporates deformable convolution DCNv3 into the original Bottleneck module. In the Bottleneck_DCNv3 module, the input feature map first undergoes convolution dimensionality reduction, DCNv3 module processing, and convolution dimensionality restoration before being concatenated with the input feature map to obtain the output result. The DCNv3 module learns the offset of the sampling points through additional convolutional layers, enabling the convolution kernel to adaptively adjust the sampling position according to the input features, thereby better adapting to the complex deformation features such as bending and breaking of laser stripes. The output result of the Bottleneck_DCNv3 module is concatenated with the result of the second part of the feature map after convolution, and the concatenated feature map is output after convolution.

[0053] 2) At the end of the backbone network, the original SPPF module is replaced with a lightweight dual-path attention module, which is structured to have parallel channel attention paths and spatial attention paths.

[0054] like Figure 4 As shown, in the lightweight dual-path attention module, the input feature map is copied into two copies after dimensionality reduction via convolution and input into two paths respectively. The channel attention path generates channel weight vectors through global average pooling and two fully connected layers. The first fully connected layer compresses the number of channels to 1 / 16 of the original, and the second fully connected layer restores the original number of channels. The channel attention weights are generated by activation through the Sigmoid function. The spatial attention path generates a spatial weight map through group normalization and depthwise separable convolution. The depthwise separable convolution uses a 5×5 convolution kernel to capture a larger receptive field. Finally, the spatial attention weights are generated through the Sigmoid function. The weight maps output from the two paths are fused by matrix multiplication to generate the final attention map. This attention map is multiplied with the original input feature map to achieve feature recalibration.

[0055] 3) In the feature pyramid of the neck network, the content-aware feature reorganization CARAFE upsampling operator is introduced to replace the traditional nearest neighbor interpolation upsampling, thus forming the CARAFE upsampling module.

[0056] The CARAFE upsampling module comprises two core components: an upsampling kernel prediction module and a feature reconstruction module. The upsampling kernel prediction module first performs channel compression on the input feature map, and then predicts the upsampling kernel at each location using convolutional layers. The feature reconstruction module, based on the predicted upsampling kernels, performs weighted combination of local regions of the input feature map to generate a high-resolution feature map. This content-aware mechanism enables the model to better preserve the edge details and texture features of the laser stripes during feature fusion.

[0057] 4) A multi-task learning mechanism was designed in the head network to simultaneously output the detection bounding box, feature point heatmap, and weld type classification results for the weld area. The head network consists of three detection layers, divided into three parallel branches: a detection branch, a keypoint branch, and a classification branch. The detection branch outputs the bounding box coordinates and confidence scores of the weld area. Keypoints are K feature points in the weld image manually annotated using Labelme. The keypoint branch uses Gaussian heatmap regression to output heatmaps of the K feature points, where the heatmap of each feature point is generated using a Gaussian function with a standard deviation of 2 pixels. The classification branch outputs the classification probabilities of five weld types. The three branches share a feature extraction network but have independent prediction heads, achieving feature sharing and knowledge transfer through multi-task learning. During model training, the difference between the predicted heatmap and the true Gaussian heatmap is calculated using a loss function. During model inference and application, sub-pixel precision 2D pixel coordinates of keypoints are extracted from the predicted heatmap using post-processing algorithms such as weighted centroid.

[0058] like Figure 6 The diagram shows the multi-task header and KPLoss calculation flowchart. In practice, the keypoint loss function KPLoss, which incorporates geometric constraints, is used, as follows:

[0059]

[0060] Among them, Loss KP This represents the total critical point loss; Loss Focal This represents the improved FocalLoss used for heatmap prediction; Loss Geometry λ represents the geometric constraint loss; hm =1.0 and λ geo =0.5 is the weighting coefficient for balancing the two losses. The heatmap loss is specifically expressed as follows:

[0061]

[0062] Where K is the total number of key points, that is, the number of feature points that need to be detected in each weld image. For a V-groove, 3 key points are usually labeled with Labelme, so K=3. H and W are the height and width of the key point heatmap, respectively. The heatmap is a two-dimensional confidence map of the model's prediction for each key point. Its size is usually a downsampling factor of the input image size. For example, if the input is 640×640, a downsampling of 4 times will result in a heatmap of 160×160. and Let i and j represent the actual and predicted heatmap values ​​of the k-th keypoint at pixel position (i,j) on the heatmap, respectively. It is generated by plotting a two-dimensional Gaussian distribution at the coordinates of the marked key points. The value of the two-dimensional Gaussian distribution is 1 at the center point and gradually decreases around the center point; predicting heatmap values. It is the confidence level of the model output after Sigmoid activation, ranging from 0 to 1.

[0063] Geometric constraint loss considers the spatial relationships between feature points:

[0064]

[0065] Where P is the set of keypoint pairs, containing all feature point pairs that need to maintain relative distance constraints. For a V-shaped bevel, the left and right edge points can be defined as a pair. A is the set of keypoint triples, used to constrain the angular relationship between feature points. For a V-shaped bevel, the left edge point, bottom point, and right edge point can be defined as a triple to constrain the bevel angle. Let represent the two-dimensional pixel coordinates of the actual annotations of the m-th and n-th keypoints in the keypoint pair set P, respectively. Represents Euclidean distance. These are the actual two-dimensional pixel coordinates of the three keypoints in the set of labeled keypoint triples A. These represent the coordinates of the corresponding key points predicted by the model. This represents calculating the angle formed by three points, ε= It is a numerically stable term.

[0066] During the model training phase, backpropagation and parameter optimization are continuously performed on the labeled keypoint dataset using a loss function. The core objective is to minimize the difference between the model-predicted heatmap and the actual Gaussian heatmap, while simultaneously constraining the spatial geometric relationships between keypoints. This process allows the weights of the convolutional neural network within the model to be adjusted, thereby learning a precise nonlinear mapping from the input weld seam image to the target keypoint heatmap.

[0067] The improved YOLOv8 weld feature detection model was trained using a training set of weld image data. The training parameters were set as follows: batch size 16, initial learning rate 0.001, AdamW optimizer, 150 training epochs, and Mosaic data augmentation disabled in the last 10 training epochs. Through these systematic improvements, this invention achieves accurate detection and localization of weld feature points in complex welding environments.

[0068] Step 5: Evaluate the performance of the weld feature extraction model based on the improved YOLOv8 using the validation set.

[0069] Step 6: Collect images of weld seams and laser stripes from the actual welding site, and use a trained weld spot detection model based on improved YOLOv8 to detect the weld spots. Output the bounding box position, confidence level, classification results, and accurate two-dimensional pixel coordinates of key points.

[0070] In the practical inference application phase, the fully trained model receives new weld seam images and outputs corresponding keypoint prediction heatmaps. These heatmaps can be viewed as two-dimensional probability distribution estimates of the keypoint locations by the model. Subsequently, through post-processing, specifically the weighted centroid method, the centroid of the confidence region is calculated from each prediction heatmap, thereby resolving the sub-pixel precision two-dimensional coordinates of the keypoints.

[0071] Step 7: Convert the two-dimensional pixel coordinates of the key points to three-dimensional coordinates in the robot's base coordinate system, that is, the actual three-dimensional coordinates of the weld point position.

[0072] By combining the extracted laser stripe pixel coordinates and the calibrated laser plane equation, the 3D coordinates of the feature points in the camera coordinate system are calculated using the principle of triangulation. These coordinates are then converted to the robot's end effector coordinates using the hand-eye calibration matrix. Finally, combined with the robot's current pose, the coordinates are converted to the robot's base coordinates, i.e., the actual 3D coordinates of the welding point location.

[0073] The core of this step is to intersect the extracted individual 2D pixel with the 3D laser plane obtained through laser plane calibration, thereby solving for the 3D coordinates of that pixel. The entire process can be broken down into the following two detailed steps:

[0074] Step 7.1: Using the principle of laser triangulation, the extracted two-dimensional pixel coordinates (u,v) of the key points (weld joints) are transformed into coordinates (X,v) in the camera coordinate system. C ,Y C Z C ). As Figure 5 As shown in the diagram illustrating the principle of laser triangulation, this model uses the optical center of the camera lens. Establish a camera coordinate system with the origin { - - - }, and at a distance from the origin along the camera's optical axis. Establish an equivalent pixel coordinate system for the position of focal length f. - - Let the intersection of the camera's optical axis and the pixel coordinate system be... Its coordinates are (u0, v0), and an image coordinate system is established with this point as the origin. - - Assume that the projection of a spatial point P into the pixel coordinate system is p, its pixel coordinates are (u,v), and its coordinates in the image coordinate system are (x,v). i ,y i If the actual physical size of each pixel along the x-axis and y-axis in the image coordinate system is dx, then... i and dy i Then, the pixel coordinates and image coordinates have the following linear relationship:

[0075]

[0076] When the image distance l equals the focal length f, the imaging point P f Image coordinates (x) p ,y p ) satisfy:

[0077]

[0078] Where dx and dy are the physical pixel dimensions along the x-axis and y-axis, respectively. For ease of subsequent calculations, the imaging points are normalized. When l = 1 mm, the normalized coordinates (x1, y1) of imaging point P1 satisfy:

[0079]

[0080] Rewritten in matrix form:

[0081]

[0082] This transformation matrix is ​​the camera's intrinsic parameter matrix, which realizes a linear mapping from the image coordinate system in millimeters to the pixel coordinate system in pixels, laying the theoretical foundation for the subsequent recovery of three-dimensional information from two-dimensional images.

[0083] At this point, the equation of the laser plane formed by the laser emitted by the line laser in space can be expressed in the camera coordinate system as:

[0084]

[0085] In the formula, ( , , ) is the coordinate of the point P to be measured on the object surface in the camera coordinate system; A, B, C, and D are the calibration parameters of the laser plane, which have been determined through a line laser plane calibration experiment.

[0086] The camera uses a pinhole imaging model, with pixel coordinates ( , ) and camera coordinate system , , The relationship between ) is:

[0087]

[0088] in: , It is the camera's focal length; , The coordinates of the camera's principal point are the center of the image.

[0089] Since this point lies both on the laser plane and on the camera's imaging ray, it satisfies:

[0090]

[0091] Solving for:

[0092]

[0093] Finally, the three-dimensional coordinates of the point P to be measured in the camera coordinate system are obtained:

[0094]

[0095] Step 7.2: After obtaining the 3D coordinates of the weld point in the camera coordinate system, it needs to be further transformed into 3D coordinates in the robot base coordinate system, i.e., the final required 3D coordinates of the weld point (X). W ,Y W Z W The coordinate transformation relationship between the pixel coordinates captured and recognized by the camera and the actual three-dimensional coordinates (world coordinates) of the object to be measured is as follows:

[0096]

[0097] Among them, (X) w , Y w Z w ) represents the world coordinates of the target weld point that are ultimately required, R is the rotation matrix of the extrinsic parameters, and t is the translation matrix of the extrinsic parameters. Both of these matrices have been calculated by hand-eye calibration experiments.

[0098] Based on the actual welding process requirements, the calculated three-dimensional coordinates are corrected and compensated. For example, for welds with gaps, when the wall thicknesses of the two workpieces to be welded are different, the welding torch needs to be offset a certain distance (e.g., 0.5mm) towards the side with the thinner wall thickness. Therefore, the system built using this method provides a coordinate offset value input interface to perform position offset compensation according to actual needs. Users can input the offset amounts in the X and Y directions, and the software automatically corrects the coordinates of all feature points.

[0099] Step 8: Transmit the actual three-dimensional coordinates of the weld point and the welding torch posture to the robot controller in real time, use them as path points, plan and execute the welding trajectory, and complete the automated welding operation.

[0100] Compared with existing technologies, the improved YOLOv8 model proposed in this invention achieves a significant improvement in weld seam recognition performance. Through systematic comparative experiments, under the same test dataset and hardware environment, the key performance indicators of the method of this invention are compared with those of the ordinary YOLOv8 model and traditional image processing methods as follows:

[0101] In terms of recognition success rate, the improved YOLOv8 model of this invention achieved a recognition accuracy of 95.2%, which is 16.6% higher than the 78.6% of the ordinary YOLOv8 model and nearly 30% higher than the 65.3% of the traditional image processing method. This significant improvement is mainly due to the deformable convolution mechanism and dual-path attention module introduced into the model, which enable the model to better adapt to the geometric deformation of the weld and maintain stable recognition performance in strong interference environments.

[0102] In terms of positioning accuracy, the method of this invention achieves an average absolute positioning error of 0.26 mm in the X direction and 0.18 mm in the Y direction, which is a significant improvement over the 0.42 mm in the X direction and 0.35 mm in the Y direction of the ordinary YOLOv8 model, and an even more significant improvement over the 0.58 mm in the X direction and 0.47 mm in the Y direction of traditional image processing methods. This is mainly attributed to the content-aware upsampling operator, which better preserves feature details, and the keypoint regression method based on heatmaps, which improves positioning accuracy.

[0103] Regarding anti-interference capabilities, the method of this invention exhibited the smallest performance degradation in tests simulating complex conditions such as strong arc light and smoke. When the signal-to-noise ratio decreased to 10dB, the method of this invention still maintained an 89.7% recognition success rate, while the ordinary YOLOv8 model and traditional image processing methods decreased to 62.3% and 45.1%, respectively. This demonstrates that the improvements proposed in this invention effectively enhance the robustness of the model under harsh conditions.

[0104] Regarding adaptability to multiple weld types, the method of this invention achieved an average recognition rate of 94.8% for five common weld types, significantly higher than the 79.2% of the ordinary YOLOv8 model and the 68.5% of traditional image processing methods. This verifies the effectiveness of the multi-task learning mechanism in complex welding scenarios.

[0105] Furthermore, this invention optimizes system resource allocation through innovative software algorithms. Its adaptive processing mechanism, while ensuring recognition accuracy, reduces the continuously high demands on hardware computing power, enabling the system to run stably on medium-configuration industrial computers and lowering deployment costs. The integrated system design, coupled with a user-friendly interface, achieves one-click operation from image acquisition and processing to coordinate transmission. This lowers the technical threshold for operators and provides a reliable solution for high-quality, high-efficiency robotic welding of complex workpieces by improving the automation and intelligence of the welding process.

[0106] The above description is only a preferred embodiment of the present invention and is not intended to limit the ideas of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A deep learning-based intelligent guided welding method for weld seams, characterized in that, include: Step 1: Build a welding system consisting of a six-axis industrial robot, a vision perception module, an industrial computer, a robot controller, a welding torch, and a welding execution module; The six-axis industrial robot is used to carry a welding torch and a vision perception module for precise movement. The welding torch is installed at the end of the robot, and the vision perception module is fixed to the welding torch tube. The vision perception module includes a frame and an industrial camera, a line laser, and a filter unit mounted on the frame, responsible for acquiring images of the weld seam and laser stripes. The industrial control computer is used to preprocess the weld seam and laser stripe images and extract image features to obtain the two-dimensional pixel coordinates of the weld seam. The robot controller is responsible for receiving the two-dimensional pixel coordinates from the industrial control computer, converting them to three-dimensional coordinates in the robot's base coordinate system, performing trajectory planning and motion control of the six-axis industrial robot, and providing real-time feedback on the robot's pose status to the industrial control computer. The welding execution module includes a welding power supply and a protective gas cylinder, responsible for completing the cladding welding process. Step 2: Perform system initialization and calibration; Step 3: Acquire images of the weld seam and laser stripes through the visual perception module, perform preprocessing, and divide them into training and validation sets in an 8:2 ratio; Step 4: Construct a weld feature extraction model based on the improved YOLOv8, and train the improved YOLOv8 weld feature extraction model using the training set data; Step 5: Evaluate the performance of the weld feature extraction model based on the improved YOLOv8 using the validation set; Step 6: Collect images of weld seams and laser stripes from the actual welding site, and use a trained weld spot detection model based on improved YOLOv8 to detect the weld spots. Output the bounding box position, confidence level, classification results, and accurate two-dimensional pixel coordinates of key points. Step 7: Transform the two-dimensional pixel coordinates of the key points to three-dimensional coordinates in the robot's base coordinate system, that is, the actual three-dimensional coordinates of the weld point position; Step 8: Transmit the actual three-dimensional coordinates of the weld point and the welding torch posture to the robot controller in real time, use them as path points, plan and execute the welding trajectory, and complete the automated welding operation. The weld feature extraction model based on the improved YOLOv8 makes the following improvements to YOLOv8: 1) A deformable convolution mechanism was introduced into the backbone and neck network of the YOLOv8 model, and the original C2f module was optimized into the C2f_DCNv3 module; 2) At the end of the backbone network, the original SPPF module is replaced with a lightweight dual-path attention module, which is structurally parallel to the channel attention path and the spatial attention path. 3) In the feature pyramid of the neck network, the content-aware feature reorganization CARAFE upsampling operator is introduced to replace the traditional nearest neighbor interpolation upsampling, thus forming the CARAFE upsampling module; 4) A multi-task learning mechanism was designed in the head network, which simultaneously outputs the detection box of the weld area, the feature point heatmap, and the weld type classification result; the head network includes three detection layers, which are divided into three parallel branches: detection branch, key point branch, and classification branch; Detect the bounding box coordinates and confidence level of the weld seam region in the branch output; The key points are K feature points in the weld seam image manually annotated by Labelme. The key point branch uses Gaussian heatmap regression to output heatmaps of the K feature points, where the heatmap of each feature point is generated by a Gaussian function with a standard deviation of 2 pixels. The classification branch outputs the classification probabilities of five weld seam types. The three branches share a feature extraction network but have independent prediction heads, achieving feature sharing and knowledge transfer through multi-task learning. During the model training phase, the difference between the predicted heatmap and the real Gaussian heatmap is calculated using a loss function. During the model inference and application phase, the sub-pixel precision 2D pixel coordinates of the key points are extracted from the predicted heatmap using a post-processing algorithm with weighted depth.

2. The intelligent guided welding method for weld seams based on deep learning according to claim 1, characterized in that, Step 2 specifically involves: Step 2.1: Camera calibration. A checkerboard calibration board with dimensions of 12×9 and actual grid size of 5mm×5mm is used. By changing the pose of the checkerboard calibration board relative to the visual perception module, images of the checkerboard calibration board in multiple poses are acquired. The images are imported into the MATLAB camera calibration toolbox. The pixel coordinates of all checkerboard corner points are obtained through an automatic corner point extraction algorithm. Based on the known physical dimensions of the checkerboard, the world coordinate system coordinates corresponding to the corner points are determined. Based on these corresponding pixel coordinates and three-dimensional world coordinates, the camera's intrinsic parameter matrix, extrinsic parameter matrix, and lens distortion coefficients are initially calculated using the Zhang Zhengyou calibration method to achieve camera calibration. Step 2.2: Hand-eye calibration. Determine the relative pose between the vision perception module and the robot's end effector welding torch. By controlling the welding torch, move the vision perception module to multiple different poses, capture images of the calibration board fixed in the workspace, and simultaneously record the transformation matrix of the robot's end effector coordinate system relative to the base coordinate system. W T E The transformation matrix of the calibration board coordinate system relative to the camera coordinate system in each image is extracted using a camera calibration algorithm. C T B Based on the coordinate system transformation relationship, the hand-eye equation is established, and the Tsai two-step method is used to solve the equation. The rotation matrix and translation vector are calculated sequentially to completely determine the rigid transformation relationship from the camera coordinate system to the robot end effector coordinate system. E T C Complete hand-eye calibration; Step 2.3: Laser plane calibration. Maintaining the fixed pose of the visual perception module, multiple sets of chessboard images projected with and without laser stripes are simultaneously acquired by alternately switching the line laser. The calibration board image without laser stripes is imported into the MATLAB camera calibration toolbox to obtain the camera extrinsic parameters corresponding to each set of images. Simultaneously, the Steger algorithm is used to extract the center line of the laser stripes from the images with laser stripes, and their pixel coordinates are converted to the image coordinate system to calculate the linear equation of the laser stripes in the image. Combining the obtained camera extrinsic parameters, the pixels on the center line of the laser stripes are converted to the camera coordinate system. The least squares method is used to perform plane fitting on these three-dimensional coordinate points to obtain the linear polynomial equation of the laser plane in the camera coordinate system, thus completing the high-precision calibration of the laser plane.

3. The intelligent guided welding method for weld seams based on deep learning according to claim 1, characterized in that, Step 3 specifically involves: Step 3.1: Control the six-axis industrial robot to move to the predefined scanning point so that the feature area of ​​the weld seam image is displayed on the image screen; Step 3.2: Project structured light stripes onto the weld to convert the three-dimensional geometric information of the weld into clear two-dimensional visual features; Step 3.3: While acquiring images using an industrial camera, continuously change the robotic arm angle, image display exposure time, contrast, and gain effect to acquire multiple images of various weld types, including V-type, corner joint, butt joint I-type, V-type with gap, and filler layer welds and laser stripe images. Step 3.4: Use Labelme software to label the weld point positions of the collected weld seam and laser stripe image data, and divide them into training set and validation set at an 8:2 ratio.

4. The intelligent guided welding method for weld seams based on deep learning according to claim 1, characterized in that, The weld feature extraction model based on the improved YOLOv8 includes a backbone network, a neck network, and a head network. The backbone network includes two basic convolutional modules, four C2f_DCNv3 modules, three downsampling convolutional modules based on convolution operations, and a lightweight dual-path attention module. The original weld image data is sequentially input into the first convolutional module, the second convolutional module, the first C2f_DCNv3 module, the first downsampling convolutional module, the second C2f_DCNv3 module, the second downsampling convolutional module, the third C2f_DCNv3 module, the third downsampling convolutional module, the fourth C2f_DCNv3 module, and the lightweight dual-path attention module of the backbone network for feature extraction.

5. The intelligent guided welding method for weld seams based on deep learning according to claim 1, characterized in that, In the C2f_DCNv3 module, the input feature map is convolved and then copied into two identical parts. The first part of the feature map is input into multiple sequentially connected Bottleneck_DCNv3 modules containing deformable convolutions for processing. Each Bottleneck_DCNv3 module incorporates deformable convolution DCNv3 into the original Bottleneck module. In the Bottleneck_DCNv3 module, the input feature map first undergoes convolution dimensionality reduction, DCNv3 module processing, and convolution dimensionality restoration before being concatenated with the input feature map to obtain the output result. The output result of the Bottleneck_DCNv3 module is concatenated with the result of the second part of the feature map after convolution. The concatenated feature map is then convolved and output.

6. The intelligent guided welding method for weld seams based on deep learning according to claim 1, characterized in that, In the lightweight dual-path attention module, the input feature map, after dimensionality reduction via convolution, is duplicated and input into two paths respectively. The channel attention path generates channel weight vectors through global average pooling and two fully connected layers. The first fully connected layer compresses the number of channels to 1 / 16 of the original, while the second fully connected layer restores the original number of channels. The channel attention weights are then generated by activation using the sigmoid function. The spatial attention path generates a spatial weight map through group normalization and depthwise separable convolution. The depthwise separable convolution uses a 5×5 kernel to capture a larger receptive field. Finally, the spatial attention weights are generated using the sigmoid function. The weight maps output from the two paths are fused through matrix multiplication to generate the final attention map, which is then multiplied by the original input feature map. Implement feature recalibration.

7. The intelligent guided welding method for weld seams based on deep learning according to claim 1, characterized in that, The CARAFE upsampling module consists of two core components: an upsampling kernel prediction module and a feature reconstruction module. The upsampling kernel prediction module first performs channel compression on the input feature map, and then predicts the upsampling kernel at each position through a convolutional layer. The feature recombination module performs weighted combination of local regions of the input feature map based on the predicted upsampling kernel to generate a high-resolution feature map.

8. The intelligent guided welding method for weld seams based on deep learning according to claim 1, characterized in that, The keypoint loss function KPLoss, which incorporates geometric constraints, is used as follows: Among them, Loss KP This represents the total critical point loss; Loss Focal This represents the improved Focal Loss used for heatmap prediction; Loss Geometry λ represents the geometric constraint loss; hm =1.0 and λ geo =0.5 is the weighting coefficient for balancing the two losses. The heatmap loss is specifically expressed as follows: Where K is the total number of keypoints, i.e., the number of feature points to be detected in each weld image, and H and W are the height and width of the keypoint heatmap, respectively. The heatmap is a two-dimensional confidence map of the model's prediction for each keypoint, and its size is usually a downsampling factor of the input image size. and Let i and j represent the actual and predicted heatmap values ​​of the k-th keypoint at pixel position (i,j) on the heatmap, respectively. It is generated by plotting a two-dimensional Gaussian distribution at the coordinates of the marked key points. The value of the two-dimensional Gaussian distribution is 1 at the center point and gradually decreases around the center point; predicting heatmap values. It is the confidence score of the model output after Sigmoid activation, ranging from 0 to 1; Geometric constraint loss considers the spatial relationships between feature points: Where P is the set of keypoint pairs, containing all feature point pairs that need to maintain relative distance constraints. P represents the number of keypoints in set P; A is the set of keypoint triples, used to constrain the angular relationships between feature points. It is the number of key points in set A; Let represent the two-dimensional pixel coordinates of the actual annotations of the m-th and n-th keypoints in the keypoint pair set P, respectively. Represents Euclidean distance. These are the actual two-dimensional pixel coordinates of the three keypoints in the set of labeled keypoint triples A. These represent the coordinates of the corresponding key points predicted by the model. This indicates the calculation of the angle formed by three points. It is a numerically stable term.

Images