Weld seam tracking method based on visual image semantic segmentation

Abstract

The invention provides a weld seam tracking method based on visual image semantic segmentation, which fully utilizes the specific keyhole characteristics of scanning laser and considers the dynamic characteristics of the keyhole in the welding process. Key holes and molten pools are effectively identified and segmented by adopting an adversarial neural network and visual semantic segmentation, so that the key holes are more accurately obtained; a keyhole rotation center is used as a basis for weld seam tracking, and is more accurate and stable compared with a molten pool center; according to the weld seam tracking method based on visual image semantic segmentation, weld seam tracking of scanning laser welding or scanning laser arc hybrid welding can be achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of weld tracking, and in particular to a weld tracking method based on visual image semantic segmentation. Background Art

[0002] Weld seam tracking technology plays a vital role in the welding process, which directly affects the welding quality, efficiency and cost. Weld seam tracking technology can monitor the position and shape of the weld in real time, ensure the integrity and quality of the weld during the welding process, and reduce the occurrence of welding defects. This is particularly important for welding applications that require high precision and high strength, such as aerospace, shipbuilding and automobile manufacturing. By automating and precisely controlling the welding process, weld seam tracking technology can improve welding speed and production efficiency, reduce labor costs and energy consumption. This is particularly valuable for large-scale production and industries that need to respond quickly to market demand. Weld seam tracking technology can also reduce manual intervention, reduce the labor intensity and operation risks of operators, and improve work safety. This is particularly important in industries where the working environment of the welding scene is harsh and is accompanied by dust, electromagnetic radiation, arc light, and noise pollution.

[0003] Weld seam tracking technology is the key to realizing welding automation, which allows welding robots or automated welding systems to perform precise welding without human intervention. It has a significant impact on improving welding quality, efficiency and reducing costs, and is an indispensable part of modern welding technology.

[0004] In order to realize weld tracking, there are already some methods in the prior art. For example, the Chinese patent application with publication number CN113828892A discloses a molten pool center recognition system and weld tracking method based on HDR images, which uses wavelet features and gradient information as recognition features, and uses the K-means algorithm to filter features, thereby improving the efficiency of the algorithm; by obtaining the deviation between the molten pool center and the weld, and correcting the position of the K-TIG welding gun in real time according to the deviation, weld tracking is realized, which can be used for automatic tracking of K-TIG welding. However, in the above method, since the shape of the molten pool is not fixed and changes all the time, the weld tracking is realized only by aligning the molten pool center with the weld center, which has certain limitations. Summary of the invention

[0005] In order to solve the problems existing in the prior art, this application proposes a weld tracking method based on visual image semantic segmentation.

[0006] In order to achieve the above objectives, this application proposes a weld tracking method based on visual image semantic segmentation, comprising the following steps:

[0007] Step 1: Fix the visual sensor on the welding platform so that it can move with the laser head or welding gun, and keep the relative position of the visual sensor and the laser head or welding gun unchanged; calibrate the visual sensor and measure the pixel distance L x , the corresponding actual distance is L s ;

[0008] Step 2: Continuously collect welding area images through visual sensors, extract the region of interest with a fixed resolution in the welding area images, and obtain images including the keyhole, molten pool, and welding gap / groove;

[0009] Step 3: Use a deep learning adversarial neural network to identify the keyhole and molten pool in the image, and then use a semantic segmentation network to assign different pixel values ​​to the keyhole, molten pool, welding gap / groove, and background, thereby segmenting each area;

[0010] Step 4: According to the pixel values ​​in the semantically segmented image, the keyhole and welding gap / groove areas are obtained to obtain a keyhole image and a welding gap / groove image;

[0011] Step 5: Use a contour search algorithm on the keyhole images at multiple consecutive moments to obtain the keyhole contour, and then use the centroid calculation formula to calculate the keyhole centroid for the keyhole contour to obtain the keyhole centroid coordinates at multiple consecutive moments;

[0012] Step 6: Calculate the rotation center of the keyhole mass center within a keyhole motion cycle according to the keyhole mass center coordinates at multiple consecutive moments to obtain the keyhole rotation center coordinates;

[0013] Step 7: Use an edge detection algorithm on the welding gap / groove image to obtain two edges of the welding gap / groove and obtain the coordinates of the two edges; wherein the two edges of the welding gap / groove are two vertical straight lines, and one edge corresponds to a set of edge coordinates, in which the coordinate value in the horizontal direction is fixed, and the coordinate value in the vertical direction continuously changes along the direction of the edge;

[0014] Step 8, averaging the horizontal coordinates of the two edges of the welding gap / groove to obtain the horizontal coordinate value of the center coordinate of the welding gap / groove;

[0015] Step 9, calculating the deviation value between the horizontal coordinates of the keyhole rotation center coordinates and the horizontal coordinates of the center coordinates of the welding gap / groove, where the deviation value is the pixel distance;

[0016] Step 10: According to the conversion relationship between pixel distance and actual distance, the deviation value between the horizontal coordinate in the keyhole rotation center coordinate and the horizontal coordinate in the center coordinate of the welding gap / groove is converted into the actual distance;

[0017] Step 11: According to the actual distance of the deviation value, adjust the laser head or welding gun to correct the deviation, thereby realizing weld tracking.

[0018] In some embodiments, in step 4, threshold segmentation is also used to binarize the keyhole image and the welding gap / groove image, and the obtained binary image is denoised to obtain an image with a clear keyhole outline and an image with a clear welding gap / groove edge.

[0019] In some embodiments, in step 5, the contour search algorithm uses the Canny edge detection algorithm to obtain the keyhole contour; the centroid calculation formula is:

[0020]

[0021] In the formula, f(x,y) is the grayscale value of the pixel at the (x,y) position after threshold segmentation, x and y are the horizontal and vertical coordinates of the image, respectively, and W and H are the centroid coordinates of the keyhole in the horizontal and vertical directions, respectively.

[0022] The beneficial effect of the scheme of the present application lies in that the weld tracking method based on visual image semantic segmentation makes full use of the keyhole features unique to the scanning laser and takes into account the dynamic characteristics of the keyhole during welding; it adopts adversarial neural network and visual semantic segmentation to effectively identify and segment the keyhole and the molten pool, and more accurately realizes the acquisition of the keyhole; the keyhole rotation center is used as the basis for weld tracking, which is more accurate and stable than using the molten pool center; the method involved in the present application can realize weld tracking of scanning laser welding or scanning laser arc hybrid welding. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of an adversarial neural network is shown.

[0024] Figure 2 A schematic diagram of the basic architecture of deep learning is shown. DETAILED DESCRIPTION

[0025] The specific implementation of the present application is further described below in conjunction with the accompanying drawings.

[0026] In the welding process, the keyhole is a crucial concept and plays an important role in ensuring the welding quality and the formation of the weld. A keyhole usually refers to a penetrating, keyhole-shaped cavity formed in the center of the weld due to the rapid melting and evaporation of the welding material in high-energy beam welding (such as laser welding or electron beam welding). The formation of the keyhole helps the rapid melting and mixing of the welding materials, while also increasing the welding depth, thereby improving the penetration of the weld. The stability of the keyhole is crucial to the welding quality because it directly affects the geometry and microstructure of the weld. For laser welding, laser arc hybrid welding, and swing laser arc hybrid welding, the keyhole generated by the laser can be used for positioning. Since the area occupied by the keyhole is relatively small, it is necessary to adjust the neural network to pay more attention to the keyhole to improve the recognition accuracy.

[0027] Deep learning is a subfield of machine learning that is based on learning algorithms based on artificial neural networks. Deep learning models learn complex patterns in data by mimicking the way the human brain processes information. Deep learning models contain multiple hidden layers, each containing multiple nodes, and are able to learn multi-level features of the data. Traditional machine learning methods usually require manual feature extraction, while deep learning models can automatically learn features from raw data.

[0028] Generative Adversarial Networks (GANs) are a system consisting of two competing neural networks, usually including a generator and a discriminator. The working principle of adversarial neural networks is as follows: the goal of the generator network is to generate data that is as correct as possible, such as pictures, music, or text. It receives raw data as input and tries to generate the result of correct data. The goal of the discriminator network is to determine whether the data generated by the generator is correct. It receives the generated data and the correct data as input and outputs the probability of whether the data generated by the generator is correct. The generator and the discriminator compete with each other during the training process. The generator generates more and more correct data, while the discriminator keeps learning to better distinguish whether the data is correct.

[0029] Based on the above information, this application proposes a weld tracking method based on visual image semantic segmentation, including the following steps:

[0030] Step 1: Fix the visual sensor on the welding platform so that it can move with the laser head or welding gun, and keep the relative position of the visual sensor and the laser head or welding gun unchanged; calibrate the visual sensor and measure the pixel distance L x , the corresponding actual distance is L s .

[0031] Step 2: Continuously collect welding area images through visual sensors, extract the region of interest with a fixed resolution in the welding area images, and obtain images including the keyhole, molten pool, and welding gap / groove.

[0032] Specifically, in step 1 and step 2, the visual sensors that can be used include but are not limited to high-speed cameras or industrial cameras. The sampling parameters of the visual sensors are set according to the experimental requirements to collect visual information of the welding process.

[0033] Step 3: Use deep learning adversarial neural network to identify keyholes and molten pools in the image, and then use semantic segmentation network to assign different pixel values ​​to keyholes, molten pools, welding gaps / grooves and backgrounds, so as to segment each area. Figures 1-2 shown.

[0034] Step 4: According to the pixel values ​​in the semantically segmented image, the keyhole and welding gap / groove areas are obtained to obtain a keyhole image and a welding gap / groove image.

[0035] In step 4, threshold segmentation can also be used to binarize the keyhole image and the welding gap / groove image, and the obtained binary image can be denoised (during the denoising process, erosion and dilation operations are used in sequence) to obtain an image with a clear keyhole contour and an image with a clear welding gap / groove edge.

[0036] Step 5: Use a contour search algorithm on the keyhole images at multiple consecutive moments to obtain the keyhole contour, and then use the centroid calculation formula to calculate the keyhole centroid for the keyhole contour to obtain the keyhole centroid coordinates at multiple consecutive moments.

[0037] Specifically, in this embodiment, the contour search algorithm uses the Canny edge detection algorithm to obtain the keyhole contour.

[0038] Specifically, the centroid calculation formula is:

[0039]

[0040] In the formula, f(x,y) is the grayscale value of the pixel at the (x,y) position after threshold segmentation, x and y are the horizontal and vertical coordinates of the image, respectively, and W and H are the centroid coordinates of the keyhole in the horizontal and vertical directions, respectively.

[0041] Step 6: Calculate the rotation center of the keyhole mass center within a keyhole motion cycle based on the keyhole mass center coordinates at multiple consecutive moments to obtain the keyhole rotation center coordinates.

[0042] Specifically, the calculation formula for the rotation center of the keyhole mass center is:

[0043]

[0044] Where W i and H i are the coordinates of the center of mass of a single keyhole in the horizontal and vertical directions, n represents the number of keyholes in a keyhole motion cycle, W s and H s are the horizontal and vertical coordinates of the keyhole rotation center respectively.

[0045] Step 7: Use an edge detection algorithm on the welding gap / groove image to obtain the two edges of the welding gap / groove and obtain the coordinates of the two edges; wherein the two edges of the welding gap / groove are two vertical straight lines, and one edge corresponds to a set of edge coordinates, in which the horizontal coordinate value is fixed and the vertical coordinate value changes continuously along the direction of the edge; the horizontal coordinates of the two edges are recorded as W L and W R .

[0046] Specifically, in this embodiment, the edge detection algorithm adopts the Sobel edge detection algorithm to obtain the edge of the welding gap / groove.

[0047] Step 8: average the horizontal coordinates of the two edges of the welding gap / groove to obtain the horizontal coordinate value of the center coordinate of the welding gap / groove.

[0048] Specifically, the horizontal coordinate value of the center coordinate of the welding gap / groove is:

[0049] W0=(W L +W R ) / 2.

[0050] Step 9: Calculate the deviation between the horizontal coordinates of the keyhole rotation center coordinates and the horizontal coordinates of the welding gap / groove center coordinates.

[0051] Specifically, σ x =|W0-W s |;

[0052] Where σ x It is the deviation value between the horizontal coordinate of the keyhole rotation center coordinate and the horizontal coordinate of the welding gap / groove center coordinate. The deviation value is the pixel distance.

[0053] Step 10: According to the conversion relationship between pixel distance and actual distance, the deviation value between the horizontal coordinate in the keyhole rotation center coordinate and the horizontal coordinate in the center coordinate of the welding gap / groove is converted into the actual distance.

[0054] Specifically, the formula for converting the deviation pixel distance to the actual distance is:

[0055]

[0056] Where σ s is the actual distance of the deviation value, σ x is the deviation pixel distance, L s and L x is the calibration parameter in step 1.

[0057] Step 11: According to the actual distance of the deviation value, adjust the laser head or welding gun to correct the deviation, thereby realizing weld tracking.

[0058] The weld tracking method based on visual image semantic segmentation involved in the present application makes full use of the keyhole features unique to the scanning laser and takes into account the dynamic characteristics of the keyhole during the welding process; it uses adversarial neural networks and visual semantic segmentation to effectively identify and segment the keyhole and the molten pool, and more accurately obtains the keyhole; the keyhole rotation center is used as the basis for weld tracking, which is more accurate and stable than using the molten pool center; the method involved in the present application can realize weld tracking of scanning laser welding or scanning laser arc hybrid welding, and the welding methods are butt welding without groove and butt welding with groove.

[0059] Example 1

[0060] This embodiment proposes a weld tracking method based on visual image semantic segmentation, in which the weld is tracked during the scanning laser welding process of aluminum alloy.

[0061] The preliminary preparation process is as follows: design the scanning laser welding experimental parameters such as laser power, welding speed, swing amplitude, swing frequency, etc.; then clamp the aluminum alloy welding specimen on the workbench in a docking manner; fix the high-speed camera on the welding platform, move with the laser head, and set the actual sampling frequency of the high-speed camera to 3000Hz; adjust the shooting area to the welding area, and then start the welding process. The weld tracking method based on visual image semantic segmentation involved in this patent application is used to track the weld in the above welding process, which can adjust the position of the welding laser head in real time so that the laser weld is always kept on the welding butt gap to achieve weld tracking.

[0062] Example 2

[0063] This embodiment proposes a weld tracking method based on visual image semantic segmentation, in which the weld is tracked during the AH36 steel scanning laser-MAG hybrid welding process.

[0064] The preliminary preparation process is as follows: design the scanning laser-MAG composite welding experimental parameters such as laser power, welding speed, swing amplitude, swing frequency, welding current, etc.; open a V-shaped groove on the AH36 steel welding specimen, and clamp the steel plate on the workbench in a butt-jointed manner. Fix the high-speed camera to the welding platform, move with the laser head and welding gun, and set the actual sampling frequency of the high-speed camera to 3000Hz; adjust the shooting area to the welding area, and then start the welding process. The weld tracking method based on visual image semantic segmentation involved in this patent application is used to track the welds in the above welding process. The position of the welding laser head and welding gun can be adjusted in real time, so that the laser arc composite welding weld is always kept on the center line of the groove, thereby realizing weld tracking.

[0065] The above is only a preferred specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can make equivalent substitutions or changes according to the technical solution and concept of the present application within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A weld tracking method based on visual image semantic segmentation, characterized in that: The following steps are involved: Step 1: Fix the visual sensor on the welding platform so that it can move with the laser head or welding gun, and keep the relative position of the visual sensor and the laser head or welding gun unchanged; calibrate the visual sensor and measure the pixel distance L x , the corresponding actual distance is L s ; Step 2: Continuously collect welding area images through visual sensors, extract the region of interest with a fixed resolution in the welding area images, and obtain images including the keyhole, molten pool, and welding gap / groove; Step 3: Use a deep learning adversarial neural network to identify the keyhole and molten pool in the image, and then use a semantic segmentation network to assign different pixel values ​​to the keyhole, molten pool, welding gap / groove, and background, thereby segmenting each area; Step 4: According to the pixel values ​​in the semantically segmented image, the keyhole and welding gap / groove areas are obtained to obtain a keyhole image and a welding gap / groove image; Step 5: Use a contour search algorithm on the keyhole images at multiple consecutive moments to obtain the keyhole contour, and then use the centroid calculation formula to calculate the keyhole centroid for the keyhole contour to obtain the keyhole centroid coordinates at multiple consecutive moments; Step 6: Calculate the rotation center of the keyhole mass center within a keyhole motion cycle according to the keyhole mass center coordinates at multiple consecutive moments to obtain the keyhole rotation center coordinates; Step 7: Use an edge detection algorithm on the welding gap / groove image to obtain two edges of the welding gap / groove and obtain the coordinates of the two edges; wherein the two edges of the welding gap / groove are two vertical straight lines, and one edge corresponds to a set of edge coordinates, in which the coordinate value in the horizontal direction is fixed, and the coordinate value in the vertical direction continuously changes along the direction of the edge; Step 8, averaging the horizontal coordinates of the two edges of the welding gap / groove to obtain the horizontal coordinate value of the center coordinate of the welding gap / groove; Step 9, calculating the deviation value between the horizontal coordinates of the keyhole rotation center coordinates and the horizontal coordinates of the center coordinates of the welding gap / groove, where the deviation value is the pixel distance; Step 10: According to the conversion relationship between pixel distance and actual distance, the deviation value between the horizontal coordinate in the keyhole rotation center coordinate and the horizontal coordinate in the center coordinate of the welding gap / groove is converted into the actual distance; Step 11: According to the actual distance of the deviation value, adjust the laser head or welding gun to correct the deviation, thereby realizing weld tracking.

2. The weld seam tracking method based on visual image semantic segmentation according to claim 1, characterized in that: In step 4, threshold segmentation is also used to binarize the keyhole image and the welding gap / groove image, and the obtained binary image is subjected to noise reduction to obtain an image with a clear keyhole outline and an image with a clear welding gap / groove edge.

3. The weld tracking method based on visual image semantic segmentation according to claim 1, characterized in that: In step 5, the contour search algorithm uses the Canny edge detection algorithm to obtain the keyhole contour; the centroid calculation formula is: In the formula, f(x,y) is the grayscale value of the pixel at the (x,y) position after threshold segmentation, x and y are the horizontal and vertical coordinates of the image, respectively, and W and H are the centroid coordinates of the keyhole in the horizontal and vertical directions, respectively.

Images