A deep learning-based robust positioning method for fine and narrow welds

Abstract

This invention discloses a robust positioning method for narrow weld seams based on deep learning. By adding an ambient light source to the weld seam tracking system and projecting it onto the plate to be welded, it overcomes the shortcomings of traditional weld seam tracking systems where laser sensors produce dark images and have low contrast during image acquisition. This provides more information-rich images suitable for the positioning algorithm proposed in this invention. The positioning algorithm proposed in this invention utilizes the powerful feature representation capabilities of convolutional neural networks and the end-to-end inference process to improve the robustness of online identification of narrow weld seams and simplify the on-site deployment process.

Description

Technical Field

[0001] This invention relates to the technical field of automated welding, and more particularly to a robust positioning method for narrow weld seams based on deep learning. Background Technology

[0002] With the continuous improvement of my country's industrialization level, automated welding accounts for an increasingly larger proportion of industrial mass production compared to manual welding. Currently, mainstream welding robots process light stripe images in real time and reconstruct three-dimensional contour data. Targeted bevel detection algorithms are developed in the contour data to locate the weld bevel, enabling automated welding operations.

[0003] Specifically, laser vision sensors are mounted on welding robots to scan the weld seam, projecting laser light onto the surface of the workpiece using active vision to reconstruct the three-dimensional contour data of the weld seam. Based on this contour data, a targeted bevel detection algorithm is developed to locate the weld seam. This method has been applied to welding many medium and large bevel workpieces and is relatively mature. However, when this processing algorithm is applied to narrow weld seams, the laser stripes do not deform significantly. Therefore, weld seam detection methods based on conventional laser vision sensors are not suitable for narrow weld seam guided welding applications. At the same time, traditional image processing algorithms for extracting regions of interest and feature points in weld seams are often ineffective when dealing with narrow weld seams of various plates under complex lighting conditions due to the uncertainty of the welding environment. In actual welding operations, when welding equipment is deployed to a new production environment or used to weld different weld seams, operators need to spend a lot of time adjusting the hyperparameters of the feature extraction algorithm due to the different characteristics of the welding target. Furthermore, the detection effect is significantly affected by ambient lighting, sometimes requiring the customization of new bevel detection algorithms. On-site equipment deployment is both time-consuming and labor-intensive.

[0004] Therefore, whether a positioning method suitable for narrow welds can be developed has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, the present invention provides a robust positioning method for narrow weld seams based on deep learning to achieve accurate positioning of narrow weld seams.

[0006] The technical solution provided by this invention is specifically a robust positioning method for narrow weld seams based on deep learning, which includes the following steps:

[0007] S1: An ambient light source is added to the weld seam tracking system. The ambient light source is projected onto the plate to be welded to ensure that the weld seam and the laser stripes projected by the weld seam tracking are visible at the same time.

[0008] S2: Acquire real-time images of the weld and enhance the images to increase the grayscale differences between the background, weld, and laser stripes. Input the processed image into a deep learning network model to perform pixel-level semantic segmentation of the three regions: background, weld, and laser stripes.

[0009] S3: After refining the segmented weld seam and laser stripe respectively, extract the center line of the weld seam and laser stripe;

[0010] S4: Based on the center lines of the weld and the laser stripe, fit and solve the intersection of the weld and the laser stripe, and calculate the pixel coordinates of the weld feature points;

[0011] S5: After smoothing the pixel coordinates of the weld feature points in step S4 using Kalman filtering, the final positioning is obtained.

[0012] Preferably, the deep learning network model in step S2 is the SP-ENet network model;

[0013] The SP-ENet network model is obtained by replacing the RegularBottleneck module in the ENet network structure with the SP-RegularBottleneck module.

[0014] Specifically, the SP-RegularBottleneck module is as follows:

[0015] 1) The input feature map is processed by 1×1 convolution, convolution, and 1×1 convolution in sequence to obtain the first feature map;

[0016] 2) The first feature map is sequentially subjected to vertical pooling, 3×1 convolution, and expansion to obtain the second feature map;

[0017] 3) Perform horizontal pooling, 1×3 convolution, and expansion on the first feature map sequentially to obtain the third feature map;

[0018] 4) After fusing the second and third feature maps, perform 1×1 convolution and activation function processing sequentially to obtain the fourth feature map;

[0019] 5) After fusing the first feature map and the fourth feature map, perform regularization processing to obtain the fifth feature map;

[0020] 6) After performing max pooling and padding on the input feature map in sequence, the sixth feature map is obtained;

[0021] 7) The fifth feature map and the sixth feature map are fused to obtain the output feature map.

[0022] Further optimized, in step S4, based on the center lines of the weld and the laser stripes, the intersection points of the weld and the laser stripes are fitted and solved to calculate the pixel coordinates of the weld feature points, specifically:

[0023] S401: After randomly and consistently sampling the center points on the center lines of the weld and the laser stripe, use the least squares method to perform line fitting, solve for the intersection of the fitted line of the weld and the fitted line of the laser stripe, and obtain the coarse positioning coordinates.

[0024] S402: Perform local resampling near the coarse positioning coordinates, use the least squares method to fit a straight line, solve for the intersection of the weld seam fitting line and the laser stripe fitting line, and obtain the precise positioning, i.e., the pixel coordinates of the weld seam feature points.

[0025] Further preferably, the ambient light source is a blue light source with the same wavelength as the laser emitter in the weld seam tracking system.

[0026] The robust positioning method for narrow weld seams based on deep learning provided by this invention overcomes the shortcomings of traditional weld seam tracking systems, such as dark images and low contrast, by adding an ambient light source to the weld seam tracking system and projecting it onto the plate to be welded. This method is more suitable for the positioning algorithm proposed in this invention. The positioning algorithm proposed in this invention utilizes the powerful feature representation capability and end-to-end reasoning process of convolutional neural networks to improve the robustness of online identification of narrow weld seams and simplify the on-site deployment process.

[0027] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit the disclosure of the present invention. Attached Figure Description

[0028] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 A flowchart illustrating a robust positioning method for narrow weld seams based on deep learning, as provided in an embodiment of the present invention.

[0031] Figure 2 The diagram shows the structure of the SP-RegularBottleneck module in a deep learning-based robust positioning method for narrow weld seams, as provided in the embodiments of the present invention.

[0032] Figure 3 This is a schematic diagram of strip pooling operation in a robust positioning method for narrow weld seams based on deep learning, provided in an embodiment of the present invention.

[0033] Figure 4 A flowchart illustrating a robust positioning method for narrow weld seams based on deep learning using a two-step positioning approach, as provided in an embodiment of the present invention.

[0034] Figure 5 Image processing for weld seams, where (a) is the image acquired by the sensor, (b) is the network segmentation result, (c) is the centerline refinement result, (d) is random consistency sampling denoising, (e) is coarse localization of weld seam feature points, and (f) is fine localization of weld seam feature points;

[0035] Figure 6 The results are the iterative results of the network model, where (a) is the iterative result of the ENet network and (b) is the iterative result of the SP-ENet network.

[0036] Figure 7 This is a comparison chart of coarse and fine positioning errors;

[0037] Figure 8 This is a comparison chart of the actual trajectory, the fine positioning trajectory, and the weld trajectory after Kalman filtering. Detailed Implementation

[0038] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of methods consistent with some aspects of the invention as detailed in the appended claims.

[0039] Previous methods for extracting regions of interest and feature points in welds based on traditional image processing algorithms often fail to perform well when dealing with narrow welds in various materials under complex lighting conditions due to the uncertainty of the welding environment. Therefore, in order to achieve accurate identification and positioning of narrow welds, this implementation scheme has made improvements in both hardware and software, thereby achieving robust positioning of narrow welds.

[0040] In terms of hardware, traditional weld seam tracking systems consist of a welding robot, control cabinet, line laser sensor, welding torch, and computer. These systems employ active vision for online guidance. The laser sensor acquires weld seam images and transmits them to the computer for weld point positioning. The control cabinet calculates the offset based on the calculated weld seam position and the current welding torch position, returning the result to the robot controller. The controller then guides the robot to correct the welding torch position and complete the welding. Currently, the detection and positioning of conventional bevel welds relies on processing laser stripes to detect feature points. While the laser stripes exhibit distinct geometric features due to the modulation of the weld bevel, the visibility of the weld area is not high. However, for narrow weld seams, the laser stripes do not produce significant geometric deformation due to the tight fit of the weld bevel. This implementation adds an ambient light source to the bottom of the laser sensor in the weld seam tracking system. Under the illumination of this ambient light source, both the weld seam and the laser stripes are simultaneously visible in the acquired image. Typically, a 450nm blue light source with the same wavelength as the laser emitter is used as the ambient light source.

[0041] Combining the hardware improvements described above, this implementation scheme provides a robust deep learning-based method for locating narrow weld seams. (See [link to relevant documentation]). Figure 1 It includes the following steps:

[0042] S1: Add an ambient light source to the weld seam tracking system. The ambient light source is projected onto the plate to be welded to ensure that the weld seam and the laser stripe projected by the weld seam tracking are visible at the same time.

[0043] S2: Acquire real-time images of the weld and enhance the images to increase the grayscale differences between the background, weld, and laser stripes. Input the processed image into a deep learning network model to perform pixel-level semantic segmentation of the three regions: background, weld, and laser stripes.

[0044] S3: After refining the segmented weld seam and laser stripe respectively, extract the center line of the weld seam and laser stripe;

[0045] S4: Based on the center lines of the weld and the laser stripe, fit and solve the intersection of the weld and the laser stripe, and calculate the pixel coordinates of the weld feature points;

[0046] S5: After smoothing the pixel coordinates of the weld feature points in step S4 using Kalman filtering, the final positioning is obtained.

[0047] Weld seam segmentation network:

[0048] In the above implementation scheme, the deep learning network model used for pixel-level semantic segmentation of the background, weld seam, and laser stripe regions is the SP-ENet (Strip Pooling-ENet) network model. This model has the same overall structure as the ENet network model, but by improving the main convolutional bottleneck block in ENet and pruning the model, it ultimately improves the segmentation accuracy while ensuring real-time performance. The SP-ENet network structure information designed in this implementation scheme is shown in Table 1.

[0049] Table 1: SP-ENet network structure when the input image resolution is 640×480

[0050]

[0051] In the network, Stages 1-4 are the encoding part, responsible for extracting image features; Stages 5-6 are the decoding part, responsible for restoring image details. Compared with the original ENet network, the SP-ENet network uses the improved SP-RegularBottleneck module in this implementation scheme for all main convolutional bottleneck modules in Stages 2 to 4 of the encoding stage.

[0052] The SP-RegularBottleneck module, see [link / reference]. Figure 2 Specifically:

[0053] 1) The input feature map is processed by 1×1 convolution, convolution, and 1×1 convolution in sequence to obtain the first feature map;

[0054] 2) The first feature map is sequentially subjected to vertical pooling, 3×1 convolution, and expansion to obtain the second feature map;

[0055] 3) Perform horizontal pooling, 1×3 convolution, and expansion on the first feature map sequentially to obtain the third feature map;

[0056] 4) After fusing the second and third feature maps, perform 1×1 convolution and activation function processing sequentially to obtain the fourth feature map;

[0057] 5) After fusing the first feature map and the fourth feature map, perform regularization processing to obtain the fifth feature map;

[0058] 6) After performing max pooling and padding on the input feature map in sequence, the sixth feature map is obtained;

[0059] 7) The fifth feature map and the sixth feature map are fused to obtain the output feature map.

[0060] This implementation improves the Regular Bottleneck module in the ENet network model by fully utilizing the long and narrow characteristics of the segmented target objects (weld regions and laser stripe regions). While standard spatial pooling operations can be used to collect spatial context information in network model design, they inevitably include a large number of irrelevant regions when dealing with irregularly shaped objects. Compared to spatial pooling that relies on square kernels, strip pooling operations more easily establish long-term dependencies between discretely distributed regions, such as... Figure 3 As shown, due to the narrow and long kernel, strip pooling also possesses the ability to capture local details with small convolutional kernels. Therefore, after the main convolution of the auxiliary branch of Regular Bottleneck, vertical and horizontal pooling operations are performed on the extracted new feature maps, respectively. Then, the pooling results are convolved and expanded, and finally, the two are added and fused, followed by 1×1 convolution. Through the multiplication operation with the input features, the relationship between each position in the input tensor and each position in the output tensor is established.

[0061] Weld feature point location:

[0062] Here, the intersection of the weld seam and the laser stripe centerline is defined as the bevel feature point. First, the centerline is extracted from the ROI region of the weld seam and laser stripe in the segmented image. Then, the intersection point is fitted to calculate the relatively accurate pixel coordinates of the weld seam feature point. The extraction of the centerline is not the core content of this invention and will not be described in detail here. In order to improve the calculation accuracy, a two-step solution approach of coarse positioning and fine positioning is proposed when calculating the pixel coordinates of the weld seam feature point.

[0063] At this point, in step S4, based on the center lines of the weld and the laser stripes, the intersection points of the weld and the laser stripes are fitted and solved to calculate the pixel coordinates of the weld feature points, specifically:

[0064] S401: After randomly and consistently sampling the center points on the center lines of the weld and the laser stripe, use the least squares method to perform line fitting, solve for the intersection of the fitted line of the weld and the fitted line of the laser stripe, and obtain the coarse positioning coordinates.

[0065] S402: Perform local resampling near the coarse positioning coordinates, use the least squares method to fit a straight line, solve for the intersection of the weld seam fitting line and the laser stripe fitting line, and obtain the precise positioning, i.e., the pixel coordinates of the weld seam feature points.

[0066] For the overall process of weld positioning based on this two-step solution approach, please refer to [link / reference]. Figure 4 .

[0067] To ensure the real-time performance of the segmentation network and its generalization ability to segment weld images under various complex environments, the trained network cannot achieve zero error. For example... Figure 5 In (b), the semantic segmentation results contain some noise due to the influence of manual spot welding. To eliminate the error caused by the fitting of interference point pairs, a random sampling consensus algorithm is used to denoise the center points.

[0068] In this phase, random consistency sampling is used to remove outliers from the set of center points of the weld and laser stripes, such as... Figure 5 In the marked part (c), each iteration randomly selects two points from the refined center point set and fits a defined straight line, as shown in formula (1). Based on the fitted straight line and the set distance threshold, the overall point set is divided into interior points and exterior points, as shown in formula (2). Finally, the set of interior points with the largest number is obtained as the final sampling result, denoted as WP. r and SP r WP r SP represents the set of sampled weld center points. r The image features after random consistency sampling of the laser stripe center point set are as follows: Figure 5 As shown in (d).

[0069]

[0070]

[0071]

[0072] The least squares method is used to fit the sampled weld center point set and the laser stripe center point set, and the intersection of the fitted lines is solved to obtain the coarse positioning result of the weld feature points. However, the actual weld shape is not an ideal straight line, and there is a certain error between the coarse positioning result and the actual weld feature points. Figure 5 (e). To further improve positioning accuracy, in the point set WP r and SP r Based on this, the center point located within 100 pixels up, down, left, and right of the coarse positioning coordinates is refitted to eliminate the influence of distant points on feature point solving, thereby improving the accuracy of the positioning results. The fine positioning results are as follows: Figure 5 As shown in (f).

[0073] Experimental results and analysis

[0074] 1) Data Acquisition

[0075] To acquire weld seam image data, a robotic welding system platform was built, using a Yaskawa GP12 robot and an IGV080 laser sensor from Beijing Tongzhou Xingye Technology Co., Ltd. To verify the segmentation effect of the designed deep learning network and the performance of the feature point extraction algorithm, 250 images of narrow butt weld seams were collected for training the network model. Simultaneously, a 33-second weld seam detection video was captured at 15fps for testing the localization algorithm.

[0076] 2) Weld seam segmentation network training

[0077] To improve the generalization of the training network, the training images were expanded to 1000 images using data augmentation techniques such as image translation, rotation, scaling, and brightness adjustment. The LabelMe annotation tool was used to annotate the background, weld seams, and laser stripe areas in the images. Finally, the annotated images were divided into training, validation, and test sets in a 7:2:1 ratio.

[0078] Since the number of pixels in the weld seam and laser stripe regions in each captured image frame is smaller than that in the background class, a multi-class cross-entropy loss function with class weights is used during training. Based on the analysis of the dataset, the class weights are set to weight = [1.4414, 36.1888, 33.0642]. To prevent overfitting, a Dropout operation with a scale of 0.1 is performed on the feature map before the auxiliary branch inference of the bottleneck structure ends. The network uses an exponentially decaying SDG optimizer, and the MIou (Mean Intersection over Union) metric is used to evaluate the network's segmentation performance. Specific network training parameter settings are shown in Table 2.

[0079] Table 2 Network Training Parameter Settings

[0080]

[0081] To compare the performance of the improved network, we also trained the segmentation network ENet and tested the inference speed of the two models on a GPU. The training results are shown in Table 3. The experimental results show that the network SP-ENet designed in this implementation scheme requires less computation and has a higher MIou index compared to ENet. The curves of loss and MIou index changes during training are shown below. Figure 6 As shown.

[0082] Table 3 Network training results

[0083]

[0084] 3) Localization Algorithm Experiment

[0085] To verify the performance of the complete localization algorithm, the trained semantic segmentation network SP-ENet was used to segment and infer each frame of the weld seam tracking video. Furthermore, the proposed two-stage localization method was employed to locate the weld seam bevel position. In the experiment, the labeled weld seam position was used as the true trajectory coordinates, and the error between the true coordinates and the detected coordinates was calculated, such as... Figure 7 As shown in the figure, it can be seen that because the coarse positioning uses all weld center points as samples in the fitting stage, the fitted straight line cannot accurately represent the weld trajectory, resulting in a large error in the coarse positioning stage. The sample points used in the fine positioning fitting are generally located around the feature points, and the local refitting method eliminates the error caused by remote data in solving for feature points. Ignoring the labeling error of the spot weld interval, most positioning errors can be controlled within 2 pixels. The positioning error statistics are shown in Table 4.

[0086] Table 4. Statistics of Positioning Errors

[0087]

[0088] When the localization algorithm processes continuous images, the weld trajectory after fine localization exhibits fluctuations due to the small movement distance of weld feature points between adjacent frames and the inherent errors in the localization algorithm itself. Therefore, a Kalman filter algorithm is used to further smooth the fine localization results, and the filtering results are quantitatively analyzed. Experiments show that the smoothness index improves from 0.706 to 0.08. The weld trajectories before and after filtering are shown below. Figure 8 .

[0089] The deep learning-based robust localization method for narrow weld seams provided by the above implementation scheme can adapt to various complex environments, has high robustness in online recognition, and adopts an improved segmentation network model. The segmentation of the weld seam region has fewer burrs, less noise, and good continuity. Furthermore, the segmentation target is fine, and the overall segmentation accuracy is high. This alleviates the disadvantage of high deployment cost in welding production. The algorithm deployment does not require the setting of a large number of hyperparameters, which simplifies the deployment process. The overall localization results are more stable, and the weld seam recognition trajectory is smoother.

[0090] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0091] It should be understood that the present invention is not limited to the content already described above, and various modifications and changes can be made without departing from its scope. The scope of the present invention is limited only by the appended claims.

Claims

1. A robust positioning method for narrow weld seams based on deep learning, characterized in that, The method includes the following steps: S1: An ambient light source is added to the weld seam tracking system. The ambient light source is projected onto the plate to be welded to ensure that the weld seam and the laser stripes projected by the weld seam tracking are visible at the same time. S2: Acquire real-time images of the weld and enhance the images to increase the grayscale differences between the background, weld, and laser stripes. Input the processed image into a deep learning network model to perform pixel-level semantic segmentation of the three regions: background, weld, and laser stripes. S3: After refining the segmented weld seam and laser stripe respectively, extract the center line of the weld seam and laser stripe; S4: Based on the center lines of the weld and the laser stripe, fit and solve the intersection of the weld and the laser stripe, and calculate the pixel coordinates of the weld feature points; S5: After smoothing the pixel coordinates of the weld feature points in step S4 using Kalman filtering, the final positioning is obtained; In step S2, the deep learning network model is the SP-ENet network model; The SP-ENet network model is obtained by replacing the RegularBottleneck module in the ENet network structure with the SP-RegularBottleneck module. Specifically, the SP-RegularBottleneck module is as follows: 1) The input feature map is sequentially subjected to 1×1 convolution for dimensionality reduction, main convolution for feature extraction, and 1×1 convolution for dimensionality increase to obtain the first feature map; 2) The first feature map is sequentially subjected to vertical pooling, 3×1 convolution, and expansion to obtain the second feature map; 3) The first feature map is sequentially subjected to horizontal pooling, 1×3 convolution, and expansion to obtain the third feature map; 4) After fusing the second and third feature maps, perform 1×1 convolution and activation function processing sequentially to obtain the fourth feature map; 5) After fusing the first feature map and the fourth feature map, perform regularization processing to obtain the fifth feature map; 6) After performing max pooling and padding on the input feature map in sequence, the sixth feature map is obtained; 7) Fuse the fifth feature map and the sixth feature map to obtain the output feature map.

2. The robust positioning method for narrow weld seams based on deep learning according to claim 1, characterized in that, In step S4, based on the center lines of the weld and the laser stripes, the intersection points of the weld and the laser stripes are fitted and solved to calculate the pixel coordinates of the weld feature points, specifically: S401: After randomly and consistently sampling the center points on the center lines of the weld and the laser stripe, use the least squares method to perform line fitting, solve for the intersection of the fitted line of the weld and the fitted line of the laser stripe, and obtain the coarse positioning coordinates. S402: Perform local resampling near the coarse positioning coordinates, use the least squares method to fit a straight line, solve for the intersection of the weld seam fitting line and the laser stripe fitting line, and obtain the precise positioning, i.e., the pixel coordinates of the weld seam feature points.

3. The robust positioning method for narrow weld seams based on deep learning according to claim 1, characterized in that, The ambient light source is a blue light source with the same wavelength as the laser emitter in the weld seam tracking system.

Images