A Weld Seam Tracking Method Based on Visual Image Semantic Segmentation

Abstract

This invention proposes a weld seam tracking method based on visual image semantic segmentation. This method fully utilizes the keyhole features unique to scanning lasers and considers the dynamic characteristics of the keyhole during welding. It employs adversarial neural networks and visual semantic segmentation to effectively identify and segment the keyhole and molten pool, achieving more accurate keyhole acquisition. Using the keyhole rotation center as the basis for weld seam tracking is more accurate and stable than using the molten pool center. The weld seam tracking method based on visual image semantic segmentation involved in this application can realize weld seam tracking in scanning laser welding or scanning laser-arc hybrid welding.

Description

Technical Field

[0001] This invention relates to the field of weld seam tracking technology, and in particular to a weld seam tracking method based on visual image semantic segmentation. Background Technology

[0002] Weld seam tracking technology plays a crucial role in the welding process, directly impacting welding quality, efficiency, and cost. It enables real-time monitoring of the weld seam's position and shape, ensuring weld integrity and quality while minimizing defects. This is particularly important for high-precision and high-strength welding applications, such as aerospace, shipbuilding, and automotive manufacturing. By automating and precisely controlling the welding process, weld seam tracking technology can improve welding speed and production efficiency, while reducing labor costs and energy consumption. This is especially valuable for industries with large-scale production and those requiring rapid market response. Furthermore, weld seam tracking technology reduces manual intervention, lowers operator workload and risks, and improves workplace safety. This is especially critical in industries with harsh welding environments and pollution from dust, electromagnetic radiation, arc light, and noise.

[0003] Weld seam tracking technology is key to achieving welding automation, allowing welding robots or automated welding systems to perform precise welding without human intervention. It has a significant impact on improving welding quality, efficiency, and reducing costs, and is an indispensable part of modern welding processes.

[0004] To achieve weld seam tracking, several methods already exist in the prior art. For example, Chinese patent application CN113828892A discloses a molten pool center recognition system and weld seam tracking method based on HDR images. This system uses wavelet features and gradient information as recognition features, and employs the K-means algorithm to filter features, improving algorithm efficiency. By acquiring the deviation between the molten pool center and the weld seam, and correcting the K-TIG welding torch position in real time based on the deviation, weld seam tracking is achieved and can be used for automatic tracking operations in K-TIG welding. However, in the above methods, since the shape of the molten pool is not fixed and changes constantly, relying solely on the alignment of the molten pool center with the weld seam center to achieve weld seam tracking has certain limitations. Summary of the Invention

[0005] To address the problems existing in the prior art, this application proposes a weld seam tracking method based on visual image semantic segmentation.

[0006] To achieve the above objectives, this application proposes a weld seam tracking method based on visual image semantic segmentation, comprising the following steps:

[0007] Step 1: Fix the vision sensor to the welding platform so that it can move together with the laser head or welding torch, keeping the relative position of the vision sensor and the laser head or welding torch constant; calibrate the vision sensor, measuring the pixel distance as L. x The corresponding actual distance is L. s ;

[0008] Step 2: Continuously acquire images of the welding area using a vision sensor, extract the region of interest at a fixed resolution from the welding area images, and obtain an image containing the keyhole, molten pool, and welding gap / groove.

[0009] Step 3: Use deep learning adversarial neural networks to identify keyholes and weld pools in the image, and then use a semantic segmentation network to assign different pixel values ​​to keyholes, weld pools, welding gaps / grooves and background, thereby segmenting each region;

[0010] Step 4: Based on the pixel values ​​in the semantically segmented image, obtain the keyhole and welding gap / bevel region to get the keyhole image and welding gap / bevel image;

[0011] Step 5: Use the contour search algorithm to obtain the keyhole contour from the keyhole images at multiple consecutive time points, and then use the centroid calculation formula to calculate the centroid of the keyhole to obtain the centroid coordinates of the keyhole at multiple consecutive time points.

[0012] Step 6: Based on the coordinates of the keyhole centroid at multiple consecutive moments, calculate the rotation center of the keyhole centroid within one keyhole motion cycle, and obtain the coordinates of the keyhole rotation center.

[0013] Step 7: Use an edge detection algorithm on the welding gap / groove image to obtain the two edges of the welding gap / groove and get the coordinates of the two edges; where the two edges of the welding gap / groove are two straight lines in the vertical direction, and each edge corresponds to a set of edge coordinates. In this set of edge coordinates, the coordinate values ​​in the horizontal direction are fixed, and the coordinate values ​​in the vertical direction change continuously along the direction of the edge.

[0014] Step 8: Average the horizontal coordinates of the two edges of the welding gap / groove to obtain the horizontal coordinate value of the center coordinates of the welding gap / groove.

[0015] Step 9: Calculate the deviation between the horizontal coordinate of the keyhole rotation center and the horizontal coordinate of the center of the welding gap / groove. This deviation is the pixel distance.

[0016] Step 10: Based on the conversion relationship between pixel distance and actual distance, convert the deviation between the horizontal coordinate of the keyhole rotation center coordinate and the horizontal coordinate of the welding gap / groove center coordinate into the actual distance.

[0017] Step 11: Adjust the laser head or welding torch according to the actual distance of the deviation value to correct the deviation, thereby achieving weld seam tracking.

[0018] In some embodiments, in step 4, threshold segmentation is also used to binarize the keyhole image and the welding gap / groove image, and noise reduction is performed on the obtained binarized image to obtain an image with a clear keyhole outline and an image with a clear welding gap / groove edge.

[0019] In some embodiments, in step 5, the contour search algorithm uses the Canny edge detection algorithm to obtain the keyhole contour; the centroid calculation formula is:

[0020]

[0021] In the formula, f(x,y) is the gray value of the pixel at position (x,y) after thresholding, x and y are the horizontal and vertical coordinates of the image, respectively, and W and H are the centroid coordinates of the keyhole in the horizontal and vertical directions, respectively.

[0022] The beneficial effects of this application's solution are as follows: the above-mentioned weld seam tracking method based on visual image semantic segmentation fully utilizes the keyhole features unique to scanning lasers and considers the dynamic characteristics of the keyhole during the welding process; it effectively identifies and segments the keyhole and molten pool using adversarial neural networks and visual semantic segmentation, thus achieving more accurate keyhole acquisition; using the keyhole rotation center as the basis for weld seam tracking is more accurate and stable than using the molten pool center; the method involved in this application can realize weld seam tracking in scanning laser welding or scanning laser-arc hybrid welding. Attached Figure Description

[0023] Figure 1 A schematic diagram of an adversarial neural network is shown.

[0024] Figure 2 A schematic diagram of the basic architecture of deep learning is shown. Detailed Implementation

[0025] The specific embodiments of this application will be further described below with reference to the accompanying drawings.

[0026] In welding, the keyhole is a crucial concept, playing a vital role in ensuring weld quality and weld formation. A keyhole, typically referring to a penetrating, key-shaped cavity formed in the center of the weld due to the rapid melting and evaporation of the welding material in high-energy beam welding (such as laser welding or electron beam welding), facilitates rapid melting and mixing of the welding material, while also increasing weld depth and thus improving weld penetration. The stability of the keyhole is critical to weld quality, as it directly affects the weld geometry and microstructure. For laser welding, laser-arc hybrid welding, and oscillating laser-arc hybrid welding, the laser-generated keyhole can be used for positioning. Because the keyhole occupies a relatively small area, the neural network needs to be adjusted to focus more on the keyhole, thereby improving recognition accuracy.

[0027] Deep learning is a subfield of machine learning that is based on learning algorithms for artificial neural networks. Deep learning models learn complex patterns in data by mimicking how the human brain processes information. A deep learning model contains multiple hidden layers, each with multiple nodes, enabling it to learn multi-level features of the data. Traditional machine learning methods typically require manual feature extraction, while deep learning models can automatically learn features from raw data.

[0028] Generative Adversarial Networks (GANs) are systems consisting of two competing neural networks, typically a generator and a discriminator. The generator network aims to generate data as accurately as possible, such as images, music, or text. It receives raw data as input and attempts to generate correct results. The discriminator network aims to determine whether the generator's data is correct. It receives both generated and correct data as input and outputs the probability of whether the generator's data is correct. The generator and discriminator compete with each other during training. The generator generates increasingly accurate data, while the discriminator continuously learns to better distinguish between correct and incorrect data.

[0029] Based on the above information, this application proposes a weld seam tracking method based on visual image semantic segmentation, including the following steps:

[0030] Step 1: Fix the vision sensor to the welding platform so that it can move together with the laser head or welding torch, keeping the relative position of the vision sensor and the laser head or welding torch constant; calibrate the vision sensor, measuring the pixel distance as L. x The corresponding actual distance is L. s .

[0031] Step 2: Continuously acquire images of the welding area using a vision sensor, extract the region of interest at a fixed resolution from the welding area image, and obtain an image containing the keyhole, molten pool, and welding gap / groove.

[0032] Specifically, in steps 1 and 2, the visual sensors that can be used include, but are not limited to, high-speed cameras or industrial cameras. The sampling parameters of the visual sensors are set according to the experimental requirements to collect visual information about the welding process.

[0033] Step 3: Use a deep learning adversarial neural network to identify keyholes and weld pools in the image. Then, use a semantic segmentation network to assign different pixel values ​​to keyholes, weld pools, welding gaps / grooves, and the background, thereby segmenting each region. Figures 1-2 As shown.

[0034] Step 4: Based on the pixel values ​​in the semantically segmented image, obtain the keyhole and welding gap / bevel region to get the keyhole image and welding gap / bevel image.

[0035] In step 4, threshold segmentation can be used to binarize the keyhole image and the welding gap / groove image, and noise reduction processing can be performed on the obtained binarized image (during noise reduction, erosion and dilation operations are used in sequence) to obtain an image with a clear keyhole outline and an image with a clear welding gap / groove edge.

[0036] Step 5: Use a contour search algorithm to obtain the keyhole contour from multiple consecutive keyhole images. Then, use the centroid calculation formula to calculate the centroid of the keyhole from the keyhole contour to obtain the centroid coordinates of the keyhole at multiple consecutive time points.

[0037] Specifically, in this embodiment, the contour search algorithm uses the Canny edge detection algorithm to obtain the keyhole contour.

[0038] Specifically, the formula for calculating the centroid is:

[0039]

[0040] In the formula, f(x,y) is the gray value of the pixel at position (x,y) after thresholding, x and y are the horizontal and vertical coordinates of the image, respectively, and W and H are the centroid coordinates of the keyhole in the horizontal and vertical directions, respectively.

[0041] Step 6: Based on the coordinates of the keyhole centroid at multiple consecutive moments, calculate the rotation center of the keyhole centroid within one keyhole motion cycle, and obtain the coordinates of the keyhole rotation center.

[0042] Specifically, the formula for calculating the center of rotation of the keyhole's centroid is:

[0043]

[0044] In the formula W i and H i Here, W represents the centroid coordinates of a single keyhole in the horizontal and vertical directions, respectively; n represents the number of keyholes in one keyhole movement cycle; and W represents the number of keyholes in one cycle. s and H s These are the coordinates of the keyhole rotation center in the horizontal and vertical directions, respectively.

[0045] Step 7: Apply an edge detection algorithm to the weld gap / groove image to obtain the two edges of the weld gap / groove and their coordinates. The two edges of the weld gap / groove are two vertical straight lines. Each edge corresponds to a set of edge coordinates. In this set of coordinates, the horizontal coordinate values ​​are fixed, while the vertical coordinate values ​​change continuously along the edge's direction. The obtained horizontal coordinates of the two edges are denoted as W. L and W R .

[0046] Specifically, in this embodiment, the Sobel edge detection algorithm is used to obtain the edge of the welding gap / groove.

[0047] Step 8: Average the horizontal coordinates of the two edges of the welding gap / groove to obtain the horizontal coordinate value of the center coordinates of the welding gap / groove.

[0048] Specifically, the horizontal coordinate value in the center coordinates of the welding gap / groove is:

[0049] W0 = (W L +W R ) / 2.

[0050] Step 9: Calculate the deviation between the horizontal coordinate of the keyhole rotation center and the horizontal coordinate of the center of the welding gap / groove.

[0051] Specifically, σ x =|W0-W s |;

[0052] In the formula σ x This is the deviation between the horizontal coordinate of the keyhole rotation center and the horizontal coordinate of the welding gap / groove center, and the deviation is the pixel distance.

[0053] Step 10: Based on the conversion relationship between pixel distance and actual distance, convert the deviation between the horizontal coordinate of the keyhole rotation center coordinate and the horizontal coordinate of the welding gap / groove center coordinate into the actual distance.

[0054] Specifically, the formula for converting the deviation pixel distance into the actual distance is as follows:

[0055]

[0056] In the formula σ s This is the actual distance of the deviation value, σ. x L is the pixel distance of the deviation value. s and L x These are the calibration parameters from step 1.

[0057] Step 11: Adjust the laser head or welding torch according to the actual distance of the deviation value to correct the deviation, thereby achieving weld seam tracking.

[0058] The weld seam tracking method based on visual image semantic segmentation disclosed in this application fully utilizes the keyhole features unique to scanning lasers and considers the dynamic characteristics of the keyhole during the welding process. It employs adversarial neural networks and visual semantic segmentation to effectively identify and segment the keyhole and molten pool, achieving more accurate keyhole acquisition. Using the keyhole rotation center as the basis for weld seam tracking is more accurate and stable than using the molten pool center. The method disclosed in this application can realize weld seam tracking in scanning laser welding or scanning laser-arc hybrid welding, with welding methods including butt welding without bevel and butt welding with bevel.

[0059] Example 1

[0060] This embodiment proposes a weld seam tracking method based on visual image semantic segmentation, which tracks the weld seam during the scanning laser welding process of aluminum alloy.

[0061] The preliminary preparation process is as follows: Laser welding experimental parameters such as laser power, welding speed, oscillation amplitude, and oscillation frequency are designed; then, the aluminum alloy welding sample is clamped onto the worktable in a butt-joint manner; a high-speed camera is fixed to the welding platform and moves with the laser head, with the actual sampling frequency of the high-speed camera set to 3000Hz; the shooting area is adjusted to the welding area, and then the welding process begins. The weld seam tracking method based on visual image semantic segmentation involved in this patent application is used to track the weld seam during the above welding process. This allows for real-time adjustment of the welding laser head position, ensuring the laser weld seam remains within the welding butt joint gap, thus achieving weld seam tracking.

[0062] Example 2

[0063] This embodiment proposes a weld seam tracking method based on visual image semantic segmentation, which tracks the weld seam during the scanning laser-MAG hybrid welding process of AH36 steel.

[0064] The preliminary preparation process is as follows: Laser power, welding speed, oscillation amplitude, oscillation frequency, welding current, and other scanning laser-MAG hybrid welding experimental parameters were designed; a V-shaped bevel was cut on the AH36 steel welding specimen, and the steel plate was clamped onto the worktable in a butt-joint manner. A high-speed camera was fixed to the welding platform and moved with the laser head and welding torch, with the actual sampling frequency of the high-speed camera set to 3000Hz; the shooting area was adjusted to the welding area, and then the welding process began. The weld seam tracking method based on visual image semantic segmentation involved in this patent application was used to track the weld seam during the above welding process. This method can adjust the position of the welding laser head and welding torch in real time, ensuring that the laser-arc hybrid welding weld seam always remains on the bevel centerline, thereby achieving weld seam tracking.

[0065] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in this application, based on the technical solution and concept of this application, should be included within the scope of protection of this application.

Claims

1. A weld seam tracking method based on visual image semantic segmentation, characterized in that: Includes the following steps: Step 1: Fix the vision sensor to the welding platform so that it can move together with the laser head or welding torch, keeping the relative position of the vision sensor and the laser head or welding torch constant; calibrate the vision sensor, measuring the pixel distance as L. x The corresponding actual distance is L. s ; Step 2: Continuously acquire images of the welding area using a vision sensor, extract the region of interest at a fixed resolution from the welding area images, and obtain an image containing the keyhole, molten pool, and welding gap / groove. Step 3: Use deep learning adversarial neural networks to identify keyholes and weld pools in the image, and then use a semantic segmentation network to assign different pixel values ​​to keyholes, weld pools, welding gaps / grooves and background, thereby segmenting each region; Step 4: Based on the pixel values ​​in the semantically segmented image, obtain the keyhole and welding gap / bevel regions to obtain the keyhole image and welding gap / bevel image; threshold segmentation is also used to binarize the keyhole image and welding gap / bevel image, and noise reduction is performed on the obtained binarized image to obtain an image with clear keyhole outline and an image with clear welding gap / bevel edge. Step 5: Apply a contour search algorithm to the keyhole images at multiple consecutive time points to obtain the keyhole contour. Then, calculate the keyhole centroid using the centroid calculation formula to obtain the keyhole centroid coordinates at multiple consecutive time points. The contour search algorithm uses the Canny edge detection algorithm to obtain the keyhole contour. The centroid calculation formula is: In the formula, f(x,y) is the gray value of the pixel at position (x,y) after thresholding, x and y are the horizontal and vertical coordinates of the image, respectively, and W and H are the centroid coordinates of the keyhole in the horizontal and vertical directions, respectively. Step 6: Based on the coordinates of the keyhole centroid at multiple consecutive moments, calculate the rotation center of the keyhole centroid within one keyhole motion cycle, and obtain the coordinates of the keyhole rotation center. Step 7: Use an edge detection algorithm on the welding gap / groove image to obtain the two edges of the welding gap / groove and get the coordinates of the two edges; where the two edges of the welding gap / groove are two straight lines in the vertical direction, and each edge corresponds to a set of edge coordinates. In this set of edge coordinates, the coordinate values ​​in the horizontal direction are fixed, and the coordinate values ​​in the vertical direction change continuously along the direction of the edge. Step 8: Average the horizontal coordinates of the two edges of the welding gap / groove to obtain the horizontal coordinate value of the center coordinates of the welding gap / groove. Step 9: Calculate the deviation between the horizontal coordinate of the keyhole rotation center and the horizontal coordinate of the center of the welding gap / groove. This deviation is the pixel distance. Step 10: Based on the conversion relationship between pixel distance and actual distance, convert the deviation between the horizontal coordinate of the keyhole rotation center coordinate and the horizontal coordinate of the welding gap / groove center coordinate into the actual distance. Step 11: Adjust the laser head or welding torch according to the actual distance of the deviation value to correct the deviation, thereby achieving weld seam tracking.

Images