A deep learning three-dimensional weld tracking method and device

Abstract

The application discloses a deep learning three-dimensional weld seam tracking method and device, which comprises the following steps: inputting real-time sampling images into a trained deep learning model to obtain weld joint structure types and weld feature point information; converting the weld feature point information into a welding path through a space transformation relationship obtained by calibration; driving a welding head to weld according to the welding path; stopping welding and moving the welding robot back to the original position when the end point of the weld seam is recognized or the welding time is exceeded or the cumulative movement distance exceeds a set value; the training steps of the deep learning model CenterNet and CenterTrack are modified, the network structures thereof are adjusted, and a network structure suitable for extracting weld picture features is obtained through single network structure search, so that the weld seam image is more suitable for line structure light sampling; and the weld feature extraction network is pre-trained in a self-supervision mode to obtain pre-training weights, so that the algorithm inference time is reduced on the basis of reducing the network structure complexity, and the recognition accuracy is improved.

Description

Technical Field

[0001] This invention belongs to the field of weld seam tracking, specifically a deep learning-based three-dimensional weld seam tracking method and device based on NAS search and pre-training. Background Technology

[0002] Currently, many robotic automated welding systems employ weld seam tracking technology for autonomous positioning during welding. This requires the system to identify the weld seam features of the workpiece in real time. Machine vision technology is one of the mainstream methods for achieving this goal. These methods include traditional morphological methods and more novel approaches such as machine learning. Traditional methods often require low computational resources and offer high real-time performance, but lack versatility, making them unsuitable for workpieces with complex three-dimensional structures. Machine learning methods, on the other hand, have advantages in versatility, but many algorithms require substantial computational resources, increasing hardware costs. Therefore, achieving high real-time machine vision weld seam tracking technology based on machine learning with limited computational resources is crucial. Existing technologies have conducted in-depth research on three-dimensional weld seam tracking to this end.

[0003] For example, invention patent application number CN202110877943 discloses a weld seam recognition method, device, and storage medium for welding robots. This patent uses line structured light technology for weld seam recognition. The recognition process primarily employs morphological calculation methods. First, it extracts the contour lines from the weld seam image. Then, based on different weld joint structure types and predefined contour types, it analyzes and extracts feature points representing the welding position from the contour lines, thereby determining the welding position. This invention can recognize various conventional weld seams and effectively addresses the difficulty of recognizing narrow weld seams in manufacturing.

[0004] For example, the invention patent with application number CN202011477237 discloses a weld seam tracking method and device based on independent correction deep learning. This method also uses line structured light technology, but its visual algorithm is implemented using the YOLO model in deep learning. This method can locate the weld seam position in real time under the presence of noise interference and has high robustness.

[0005] The aforementioned prior art has the following shortcomings:

[0006] (1) Weld tracking technology based on traditional morphological methods is often designed for specific weld joint structure types, such as fillet welds, butt welds, lap welds, etc., and the noise elimination mode is singular. For the identification of welds or composite welds of some special-shaped workpieces, it lacks versatility and is difficult to achieve flexible production.

[0007] (2) Currently, many methods do not have in-depth designs for the automatic determination of the start and end positions of the weld seam tracking process, and many aspects related to this also need to be implemented through some auxiliary markers. At the same time, since the images sampled by line structured light technology are two-dimensional images, it is difficult to determine the specific position of the image in the entire weld seam from the sampled images solely through morphological methods.

[0008] (3) To achieve highly flexible production, many weld seam tracking methods employ machine learning, especially deep learning technology. This type of technology requires a large amount of computing resources, which increases computing costs, reduces the real-time performance of the system, and affects welding efficiency.

[0009] (4) Currently, many deep learning methods applied in this field adopt candidate box-based network structures, such as YOLO and Faster-RCNN. However, the welding position of a weld is usually only one point. Therefore, these candidate box-based methods are not entirely suitable for identifying weld feature points and consume a lot of redundant computing resources.

[0010] (5) CenterNet is a deep learning method that does not require candidate boxes. It has a fast recognition speed and only requires the location of the recognition position. It is suitable for weld seam tracking scenarios based on line structured light. However, it does not connect image sets with temporal correlation, wasting the temporal correlation of the continuous dataset, which affects the recognition efficiency and accuracy.

[0011] To address this, the applicant's earlier invention patent application, CN202111598385, discloses a laser vision weld seam tracking system, device, and storage medium based on CenterNet. This method designs a weld seam tracking system for a welding robot equipped with line structured light based on the CenterNet vision algorithm. Furthermore, this solution integrates robot motion state information and images, enabling the system to combine visual features and robot motion states for end-to-end training, improving the real-time performance of weld seam tracking and increasing optimization potential. However, this invention also fails to link temporally correlated image sets, wasting the temporal correlation of continuously occurring datasets and affecting recognition efficiency and accuracy. Summary of the Invention

[0012] A brief overview of embodiments of the invention is provided below to provide a basic understanding of certain aspects of the invention. It should be understood that this overview is not an exhaustive summary of the invention. It is not intended to identify key or essential parts of the invention, nor is it intended to limit the scope of the invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to the more detailed description that follows.

[0013] According to one aspect of this application, a deep learning-based three-dimensional weld seam tracking method is provided, applied in a three-dimensional weld seam tracking device. The three-dimensional weld seam tracking device includes a welding robot, a robot control cabinet, a line structured light sensor, a welding machine, a welding head, and an industrial control computer. The line structured light sensor and the welding head are coaxially mounted on the end effector of the welding robot via a mounting bracket. During welding tracking, the welding robot drives the end effector to move, and the welding head and the line structured light sensor simultaneously sample and weld the workpiece placed on the worktable. The three-dimensional weld seam tracking method includes:

[0014] Step 1: The image sampled in real time by the line structured light sensor is input into the trained deep learning model to obtain the weld joint structure type and weld feature point information;

[0015] Step 2: The spatial transformation relationship obtained through calibration of weld feature point information is converted into control points to guide the robot's movement, forming a welding path, which is then sent to the welding robot;

[0016] Step 3: The welding robot moves, driving the welding head to perform welding according to the welding path.

[0017] Step 4: When the end point of the weld is identified, or the welding timeout is exceeded, or the cumulative movement distance exceeds the set value, stop welding and move the welding robot back to the origin.

[0018] The training steps for the deep learning model are as follows:

[0019] Step a: Collect weld sample images and create a dataset;

[0020] Step b: Modify the structure of the weld detection network CenterNet and the weld tracking network CenterTrack;

[0021] Step c: Perform self-supervised pre-training on the weld feature extraction network to obtain pre-trained weights;

[0022] Step d: Train the modified weld detection network CenterNet and weld tracking network CenterTrack to obtain a deep learning model.

[0023] A line structured light generator projects structured light onto the surface of the workpiece to be tested, forming a laser stripe. A CCD camera samples the laser stripe image through a filter to complete the sampling. With this technology, the system can obtain the morphological information of the workpiece surface through the shape of the laser stripe, thereby determining which feature point on the laser stripe is the location to be welded. Based on this characteristic, this method uses two deep learning networks: CenterNet and CenterTrack. The CenterNet network is responsible for locating and identifying the category and location of the object from the image. This method can treat the target as a point without using candidate boxes for identification, making it suitable for this application scenario. On the other hand, various interferences exist during the welding process, such as arc light interference and spatter interference. Therefore, the CenterTrack network is used to correlate images in time, stably identifying weld feature points in an interference environment. Furthermore, to make the network structures of CenterNet and CenterTrack suitable for weld feature recognition, a feature extraction network was searched using a single network structure search method, and pre-trained weights were obtained through self-supervised training. This modified CenterNet and CenterTrack to be more suitable for visual acquisition applications in weld tracking scenarios based on line structured light technology.

[0024] Furthermore, step (a) includes the following steps:

[0025] Step (a1): Sample four common types of weld joint structures to obtain a dataset;

[0026] Step (a2): Label the weld joint structure type and welding feature point coordinates of the dataset in step (a1);

[0027] Step (a3): Remove a portion of the workpieces from the batch of three-dimensional complex structure workpieces that need to be welded; use a teaching method to enable the robot to sample the weld along the welding path;

[0028] Step (a4): Label the coordinates of welding feature points in the dataset obtained in step (a3), and at the same time, select the images of the start and end points of the weld in the dataset as special types.

[0029] In steps (a1) and (a3), the dataset refers to the set of images sampled by the structured light sensor. During each sampling, the robot carries the sensor from one end of the weld to the other, capturing images at a certain frequency. Therefore, in each sampled image set, adjacent images are temporally correlated. Step (a2) mentions four common weld joint structure types: butt joint, inner corner joint, outer corner joint, and lap joint.

[0030] Furthermore, step (b) of modifying the structure of the weld inspection network CenterNet and the weld tracking network CenterTrack includes the following steps:

[0031] Step (b1): Remove the size prediction branch of CenterNet, and modify the loss function of the weld detection network structure to L. det =L k_d +λ off_d L off_d L k_d For heatmap prediction loss function, L off_d Let λ be the loss function for predicting the center point offset of the heatmap. off_d These are coefficients used to control the weights between the loss functions;

[0032] Step (b2): Remove the displacement prediction branch of CenterTrack, and modify the loss function of the weld tracking network structure to L. track =L k_t +λ off_t L off_t L k_t For heatmap prediction loss function, L off_t Let λ be the loss function for predicting the center point offset of the heatmap. off_t L is a coefficient used to control the weights between the loss functions; where L k_d and L k_t Same, L off_d and L off_t Same, λ off_d and λ off_t Similarly, removing the displacement prediction branch of CenterTrack results in an altered loss function L for its modified weld seam tracking network structure. track The loss function L of the weld inspection network structure modified by removing the size prediction branch of CenterNet det same;

[0033] Step (b3) modifies the steps for extracting peaks from the heatmap in CenterNet and CenterTrack, and directly outputs the maximum value of the heatmap as the identified weld location.

[0034] Step (b4): Use the single-pass network structure search method to obtain the CenterNet and CenterTrack feature extraction networks suitable for weld feature extraction.

[0035] In step (b4), the "One-shoot NAS" refers to a method called "Network Structure Search (NSA)," which is a concept of a class of methods that use automated search algorithms to find the structure of deep neural networks. One-shoot NAS is a specific method within the NSA category. The purpose of employing "One-shoot NAS" in this invention is to replace the feature extraction networks of the native CenterNet and CenterTrack networks, making the network structure more focused on detecting and tracking weld feature points that conform to specific patterns. Therefore, this invention uses weld images for training during the hypergraph training process, ensuring that the network structure in the trained hypergraph is applicable to the aforementioned scenario.

[0036] Furthermore, step (b4), which uses a single-pass network structure search method to obtain the CenterNet and CenterTrack feature extraction networks suitable for weld feature extraction, includes the following steps:

[0037] Step (b41): Design the network search space and use the moving-flipping bottleneck convolution as the basic module of the network. The entire search space includes 4 different moving-flipping bottleneck convolution modules. Each module selects the same or different parameters such as expandratio, layers, kernel size and stride.

[0038] Step (b42): Based on the designed network search space, train its hypergraph (containing all possible feature extraction networks) using the single-path uniform sampling method and the SGD gradient descent method on the weld seam image dataset until convergence;

[0039] Step (b43): Develop evaluation criteria for the network, use a genetic algorithm to evaluate each possible feature extraction network using the parameters trained in the hypergraph, and search for the optimal network structure as the weld feature extraction network.

[0040] In step (b43), the evaluation criterion is cost = err + maxerr + flops × 3, where err represents the average pixel error of the sub-network used for weld position detection, maxerr represents the maximum pixel error, and flops represents the computational cost.

[0041] Furthermore, step (c) of performing self-supervised pre-training on the weld feature extraction network to obtain pre-training weights specifically includes the following steps:

[0042] Step (c1): Use the weld feature extraction network obtained in step (b) as an encoder, and add a decoder composed of stacked deconvolution layers at the end of the network to form an encoder-decoder network structure.

[0043] Step (c2): Input the masked weld seam image into the model and use the unmasked image as a supervision signal for training.

[0044] The image of the masked weld in step (c2) refers to:

[0045] Half of the unlabeled images in the dataset are selected for pixel masking, and the other half is selected for region masking. Pixel masking refers to covering certain random pixels in the image with masking values, while region masking refers to randomly generating a rectangular area in the image composed of masking pixels.

[0046] Furthermore, step (d) of training the modified weld detection network CenterNet and weld tracking network CenterTrack specifically includes the following steps:

[0047] Step (d1): Initialize the modified weld detection network CenterNet using pre-trained weights;

[0048] Step (d2): The dataset is augmented by random horizontal rotation, addition of random noise, and random translation to obtain the augmented data;

[0049] Step (d3): Input the enhanced data into the modified weld inspection network CenterNet for iterative training;

[0050] Step (d4): Initialize the weld tracking network CenterTrack using the trained weld detection network CenterNet model parameters;

[0051] Step (d5): Combine the time-related images in the dataset into real video data, and at the same time, simulate the changes in the weld seam image during the sampling process by randomly scaling and translating some images, thereby generating a simulated video;

[0052] Step (d6): Input the real video and the simulated video into the CenterTrack weld tracking network model for training.

[0053] Furthermore, step 1 includes the following steps:

[0054] Step (11): Input an image. First, the CenterNet network identifies the type of weld and the coordinates of the welding position in the image. At the same time, it outputs a confidence score for each type. If the confidence score exceeds the threshold, the identification is considered successful; otherwise, it is considered a failure.

[0055] Step (12): After inputting a new image, based on the heat of the previous image, the current image, and the previous image... Figure 3The information is obtained by the CenterTrack network to track the coordinates of the weld feature points in the current image and the confidence level corresponding to the coordinates. Similar to step (11), if the confidence level exceeds the threshold, the tracking of this frame is considered successful; otherwise, it is considered a failure. Among them, the heat map of the first image is the same as the heat map in step (b3). Figure 1 The images refer to those obtained by performing a convolution operation on the output of the feature extraction network by the weld seam tracking network CenterNet.

[0056] Step (13): If tracking fails during the tracking process, rerun CenterNet to detect and locate the new weld feature point coordinates;

[0057] In step (12), the heatmap refers to the image obtained by convolving the output of the feature extraction network by CenterNet. Specifically, for an image I∈R with height H, length W, and 3 channels... W×H×3 The heat map is Where R is the downsampling step size for depth feature extraction, and C is the target type to be detected. In the heatmap, the coordinates of pixels with a value of 1 are the predicted coordinates of the weld seam welding position.

[0058] The heat map in step (b3) has the same meaning as the heat map in step (12).

[0059] According to another aspect of this application, a deep learning-based three-dimensional weld seam tracking device is provided, applied in a three-dimensional weld seam tracking equipment. This three-dimensional weld seam tracking equipment includes a welding robot, a robot control cabinet, a line structured light sensor, a welding machine, a welding head, and an industrial control computer. The line structured light sensor and the welding head are coaxially mounted on the end effector of the welding robot via a mounting bracket. During welding tracking, the welding robot drives the end effector to move, and the welding head and the line structured light sensor simultaneously sample and weld the workpiece placed on the worktable. The three-dimensional weld seam tracking device includes:

[0060] The first module is used to input real-time sampled images from the line structured light sensor into the trained deep learning model to obtain information on the weld joint structure type and weld feature points.

[0061] The second module is used to convert the spatial transformation relationship obtained by calibration of weld feature point information into control points that guide the robot's movement to form a welding path, and then send it to the welding robot.

[0062] The third module is used to enable the welding robot to drive the welding head to perform welding according to the welding path.

[0063] The fourth module is used to stop welding and move the welding robot back to its origin when the end point of the weld is detected, the welding timeout is exceeded, or the cumulative movement distance exceeds the set value.

[0064] The training steps for the deep learning model are as follows:

[0065] Step a: Collect weld sample images and create a dataset;

[0066] Step b: Modify the structure of the weld detection network CenterNet and the weld tracking network CenterTrack;

[0067] Step c: Perform self-supervised pre-training on the weld feature extraction network to obtain pre-trained weights;

[0068] Step d: Train the modified weld detection network CenterNet and weld tracking network CenterTrack to obtain a deep learning model.

[0069] The present invention employs the above method and has the following advantages:

[0070] 1. It can generalize and identify various weld joint structures, and specifically sample weld joint structures with complex three-dimensional structures to extend the model's ability to identify atypical weld joint structures.

[0071] 2. Utilize the CenterTrack network to improve recognition accuracy and anti-interference capabilities in complex environments;

[0072] 3. By searching the feature extraction network through a single network structure, the CenterNet and CenterTrack networks were modified to better adapt to weld seam images sampled by line structured light. This reduced the complexity of the network structure, decreased the algorithm's inference time, and improved the accuracy of recognition.

[0073] 4. By using a self-supervised training method, the difficulty of labeling the dataset used to train the weld seam recognition feature extraction network is reduced. Attached Figure Description

[0074] The present invention can be better understood by referring to the description given below in conjunction with the accompanying drawings, in which the same or similar reference numerals are used throughout the drawings to denote the same or similar parts. These drawings, together with the following detailed description, are incorporated in and form part of this specification, and are used to further illustrate preferred embodiments of the invention and explain the principles and advantages of the invention.

[0075] In the picture:

[0076] Figure 1 This is an overall structural diagram of the three-dimensional weld seam tracking device of the present invention;

[0077] Figure 2 This is a schematic diagram of the sampling of the line structure optical sensor of the present invention;

[0078] Figure 3 This is a flowchart illustrating the actual weld seam tracking process of the present invention.

[0079] Figure 4 The following are the design steps for the deep learning model of this invention. Detailed Implementation

[0080] Embodiments of the present invention will now be described with reference to the accompanying drawings. Elements and features described in one drawing or embodiment of the invention may be combined with elements and features shown in one or more other drawings or embodiments. It should be noted that, for clarity, representations and descriptions of components and processes unrelated to the present invention and known to those skilled in the art have been omitted from the drawings and description.

[0081] The technical problems to be solved by this invention include:

[0082] (1) The welds of three-dimensional sheet metal workpieces often have different shapes and complex weld joint structures. Therefore, it is necessary to generalize the identification of weld joint structures and realize the automatic classification of various weld joint structures during the welding process.

[0083] (2) It is hoped that the start and end points of the weld can be automatically identified during the weld tracking process, so as to realize the automatic start and stop of the welding process;

[0084] (3) Improve recognition accuracy by utilizing the temporal relationship between images;

[0085] (4) It can complete the training of machine learning models through self-supervision, reducing the difficulty of labeling training samples during batch welding;

[0086] (5) Optimize the feature extraction network structure by using single network structure search technology to reduce the computational load of deep learning algorithms and improve the real-time performance of the system.

[0087] The hardware used in this application is the hardware disclosed in the invention patent application number 202111467999.7, which describes a three-dimensional trajectory laser welding seam tracking attitude planning method. Figure 1 and Figure 2As shown, the system includes a welding robot 1, a robot control cabinet 2, a line structured light sensor 3, a welding machine 4, a welding head 5, an industrial computer 6, a welding table 7, and a mounting bracket 9. The workpiece 8 to be tested is fixed to the welding table 7 using a simple clamp. The line structured light sensor 3 and the welding head 5 are mounted on the mounting bracket 9, coaxially mounted at the end of the welding robot 1 via the mounting bracket 9. The line structured light sensor 3 is mounted a distance in front of the laser welding head 5 in the direction of movement. The mounting bracket 9 is mounted on the flange of the 6-axis robotic arm of the welding robot 1. The robot control cabinet 2 is responsible for the motion control of the robotic arm. The industrial computer 6 receives the sampling information from the line structured light sensor, sends motion commands to the industrial computer 2, and simultaneously controls the welding machine 4 to perform welding. During tracking welding, the robot drives the end effector to move, and the welding head and sensor simultaneously sample and weld the workpiece 8 placed on the worktable 7. Alternatively, other multi-axis motion platforms can also be used to implement this robot.

[0088] Figure 2 The schematic diagram of the line structured light sensor sampling shown is consistent with the deep learning-based laser welding seam feature point recognition method disclosed in the inventor's prior application, application number CN202111467997. Figure 3 Similarly, line structured light is an active vision technology, and its working principle is as follows: Figure 2 First, a structured light generator 14 projects structured light 13 onto the surface of the workpiece 15 to be tested, forming a laser stripe 12. A CCD camera 10 samples the laser stripe image through a filter 11 to complete the sampling. With this technology, the system can obtain the morphological information of the workpiece surface through the shape of the laser stripe, thereby determining which feature point on the laser stripe is the location to be welded. Based on this characteristic, the method of this invention uses two types of deep learning networks: CenterNet and CenterTrack. The CenterNet network is responsible for locating and identifying the category and location of the object from the image. This method can treat the target as a point without using candidate boxes for identification, making it suitable for this application scenario. On the other hand, various interferences exist during the welding process, such as arc light interference and spatter interference. Therefore, the CenterTrack network is used to correlate images in time, stably identifying weld feature points in an interference environment. Furthermore, to make the network structures of CenterNet and CenterTrack suitable for weld feature recognition, a feature extraction network was searched using a single network structure search method, and pre-trained weights were obtained through self-supervised training. This modified CenterNet and CenterTrack to be more suitable for visual acquisition applications in weld tracking scenarios based on line structured light technology.

[0089] This invention provides a deep learning-based three-dimensional weld seam tracking method based on NAS search and pre-training, including a deep learning model training process and an actual welding process;

[0090] The deep learning model training process includes:

[0091] (a) Creating a training set;

[0092] (b) Modify the network structure of the weld detection network CenterNet and the weld tracking network CenterTrack;

[0093] (c) Self-supervised pre-training is performed on the weld feature extraction network to obtain pre-training weights;

[0094] (d) Train the modified weld detection network CenterNet and weld tracking network CenterTrack to obtain a deep learning model;

[0095] After training the deep learning model, the algorithm is applied to actual welding, specifically including:

[0096] (e) The image sampled in real time by the line structured light sensor is input into the trained model to obtain the weld joint structure type and weld feature point information;

[0097] (f) Transform the spatial transformation relationship of the weld feature points obtained through calibration into control points to guide the robot's motion, and send them to the robot;

[0098] (g) The robot moves, driving the welding head to perform welding;

[0099] (h) When the end point of the weld is identified, the welding time is exceeded, or the cumulative movement distance exceeds the set value, the welding is stopped and the robot is moved back to the origin.

[0100] Step (a) specifically includes the following steps:

[0101] (a1) Sample four common types of welded joint structures to obtain a dataset;

[0102] (a2) Label the weld joint structure type and welding feature point coordinates of the dataset in step (a1);

[0103] (a3) Take out a portion of the workpieces from the batch of three-dimensional complex structure workpieces that need to be welded; use teaching to enable the robot to sample the weld along the welding path;

[0104] (a4) Label the coordinates of welding feature points in the dataset obtained in step (a3), and select the images of the start and end points of the weld in the dataset as special types.

[0105] In steps (a1) and (a3), the dataset refers to the set of images sampled by the line structured light sensor. Each time the sensor is sampled, the robot carries the sensor from one end of the weld to the other. During this time, the sensor takes pictures at a certain frequency. Therefore, in the set of images obtained in each sampling, adjacent images are temporally related.

[0106] The four common weld joint types in step (a2) are butt joint, internal corner joint, external corner joint, and lap joint.

[0107] Step (b) specifically includes the following steps:

[0108] (b1) Remove the size prediction branch from CenterNet, and the loss function of the modified weld detection network structure is L. det =L k_d +λ off_d L off_d ;

[0109] Step (b2): Remove the displacement prediction branch of CenterTrack, and modify the loss function of the weld tracking network structure to L. track =L k_t +λ off_t L off_t ;

[0110] In steps (b1) and (b2), removing the displacement prediction branch of CenterTrack leads to an increase in the loss function L of the modified weld seam tracking network structure. track The loss function L of the weld inspection network structure modified by removing the size prediction branch of CenterNet det Same, L k_d and L k_t For heatmap prediction loss function, L off_d and L off_t Let λ be the loss function for predicting the center point offset of the heatmap. off_t and λ off_d L is the coefficient used to control the weights between the loss functions. k_d and L k_t Same, L off_d and L off_t Same, λ off_d and λ off_t same;

[0111] (b3) Modify the steps for extracting peaks from the heatmap in CenterNet and CenterTrack, and directly output the maximum value of the heatmap as the identified weld location.

[0112] (b4) The CenterNet and CenterTrack feature extraction networks suitable for weld feature extraction are obtained by using the single network structure search method.

[0113] The evaluation criterion in step (b3) is cost = err + maxerr + flops × 3, where err represents the average pixel error of the sub-network used for weld position detection, maxerr represents the maximum pixel error, and flops represents the computational cost.

[0114] Step (b4) includes the following steps:

[0115] (b41) Design the network search space and use the moving flip bottleneck convolution as the basic module of the network. The entire search space includes 4 different moving flip bottleneck convolution modules. Each module can select different parameters such as expand ratio, layers, kernel size and stride.

[0116] (b42) Based on the designed search space, train its hypergraph (containing all possible feature extraction networks) using the single-path uniform sampling method and the SGD gradient descent method on the weld seam image dataset until convergence;

[0117] (b43) Develop evaluation criteria for the network, use a genetic algorithm to evaluate each possible feature extraction network using the parameters trained in the hypergraph, and search for the optimal network structure.

[0118] Step (c) includes the following steps:

[0119] (c1) Use the weld feature extraction network obtained in step (b) as an encoder, and add a decoder composed of stacked deconvolution layers at the end of the network to form an encoder-decoder network structure.

[0120] (c2) Input the masked weld seam image into the model and use the unmasked image as the supervision signal for training.

[0121] The image of the masked weld in step (c2) refers to:

[0122] Half of the unlabeled images in the dataset are selected for pixel masking, and the other half is selected for region masking. Pixel masking refers to covering certain random pixels in the image with masking values, while region masking refers to randomly generating a rectangular area in the image composed of masking pixels.

[0123] Step (d) specifically includes the following steps:

[0124] (d1) Initialize the modified weld detection network CenterNet using pre-trained weights;

[0125] (d2) The data is enhanced by random horizontal rotation, adding random noise, and random translation;

[0126] (d3) Input the enhanced data into the modified weld inspection network CenterNet for iterative training;

[0127] (d4) Initialize the weld tracking network CenterTrack using the trained weld detection network CenterNet model parameters;

[0128] (d5) Combine the time-related images in the dataset into real video data, and at the same time, simulate the change process of the weld seam image during the sampling process by randomly scaling and translating some images, thereby generating a simulated video;

[0129] (d6) Input real and simulated videos into the CenterTrack model for training.

[0130] Step (e) includes the following steps:

[0131] (e1) Input an image. First, the CenterNet network identifies the type of weld and the coordinates of the welding position in the image. At the same time, it outputs a confidence score for each type. If the confidence score exceeds the threshold, the identification is considered successful; otherwise, it is considered a failure.

[0132] (e2) After inputting a new image, the heat of the previous image, the current image, and the previous image will be used as the basis for the image's heat. Figure 3 The information is obtained by the CenterTrack network to track the coordinates of the weld feature points in the current image and the confidence level corresponding to the coordinates. Similar to step (e1), if the confidence level exceeds the threshold, the tracking of this frame is considered successful; otherwise, it is considered a failure.

[0133] (e3) If a tracking failure occurs during the tracking process, CenterNet is rerun to detect and locate the new weld feature point coordinates.

[0134] The heat map in step (e2) and the heat map in step (b3) Figure 1 Both "image" and "sample" refer to the image obtained by convolving the output of the feature extraction network using CenterNet. Specifically, for an image with height H, length W, and 3 channels... Heat map Where R is the downsampling step size for depth feature extraction, and C is the type of target to be detected. In the heatmap, the coordinates of pixels with a value of 1 are the predicted coordinates of the weld seam welding position.

[0135] This invention, through the aforementioned scheme, trains an algorithm model capable of generalizing the recognition of weld joint structures in complex 3D workpieces by sampling a targeted dataset. In weld seam tracking applications, the CenterTrack algorithm leverages the temporal relationships between adjacent frames to improve recognition accuracy and anti-interference capabilities. By modifying the feature extraction networks of CenterNet and CenterTrack through a single network structure search, it becomes suitable for weld seam feature recognition tasks based on line structured light, reducing network complexity, lowering algorithm inference time, and simultaneously improving recognition accuracy. In weld seam tracking applications, a self-supervised training method is used to train the feature extraction network, reducing the difficulty of labeling the dataset used to train the weld seam recognition feature extraction network.

[0136] This invention also provides a deep learning-based three-dimensional weld seam tracking device, applied in a three-dimensional weld seam tracking equipment. The three-dimensional weld seam tracking equipment includes a welding robot, a robot control cabinet, a line structured light sensor, a welding machine, a welding head, and an industrial control computer. The line structured light sensor and the welding head are coaxially mounted on the end effector of the welding robot via a mounting bracket. During welding tracking, the welding robot drives the end effector to move, and the welding head and the line structured light sensor simultaneously sample and weld the workpiece placed on the worktable. The invention is characterized in that the three-dimensional weld seam tracking device includes:

[0137] The first module is used to input real-time sampled images from the line structured light sensor into the trained deep learning model to obtain information on the weld joint structure type and weld feature points.

[0138] The second module is used to convert the spatial transformation relationship obtained by calibration of weld feature point information into control points that guide the robot's movement to form a welding path, and then send it to the welding robot.

[0139] The third module is used to enable the welding robot to drive the welding head to perform welding according to the welding path.

[0140] The fourth module is used to stop welding and move the welding robot back to its origin when the end point of the weld is detected, the welding timeout is exceeded, or the cumulative movement distance exceeds the set value.

[0141] The training steps for the deep learning model are as follows:

[0142] Step a: Collect weld sample images and create a dataset;

[0143] Step b: Modify the structure of the weld detection network CenterNet and the weld tracking network CenterTrack;

[0144] Step c: Perform self-supervised pre-training on the weld feature extraction network to obtain pre-trained weights;

[0145] Step d: Train the modified weld detection network CenterNet and weld tracking network CenterTrack to obtain a deep learning model.

[0146] This invention also provides a computer-readable storage medium storing a computer program that is executed by a processor to implement the methods described above.

[0147] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0148] Furthermore, the method of the present invention is not limited to being executed in the chronological order described in the specification, but may also be executed in other chronological orders, in parallel, or independently. Therefore, the execution order of the method described in this specification does not constitute a limitation on the technical scope of the present invention.

[0149] Although the invention has been disclosed above through the description of specific embodiments, it should be understood that all the embodiments and examples described above are exemplary and not restrictive. Those skilled in the art can design various modifications, improvements, or equivalents to the invention within the spirit and scope of the appended claims. These modifications, improvements, or equivalents should also be considered to be included within the protection scope of the invention.

Claims

1. A deep learning-based three-dimensional weld seam tracking method, applied in a three-dimensional weld seam tracking device, the three-dimensional weld seam tracking device comprising a welding robot, a robot control cabinet, a line structured light sensor, a welding machine, a welding head, and an industrial control computer, wherein the line structured light sensor and the welding head are coaxially mounted on the end effector of the welding robot via a mounting bracket. During tracking welding, the welding robot drives the end effector to move, and the welding head and the line structured light sensor simultaneously sample and weld the workpiece placed on the worktable; characterized in that: The three-dimensional weld seam tracking method includes: Step 1: The image sampled in real time by the line structured light sensor is input into the trained deep learning model to obtain the weld joint structure type and weld feature point information; Step 2: The spatial transformation relationship obtained through calibration of weld feature point information is converted into control points to guide the robot's movement, forming a welding path, which is then sent to the welding robot; Step 3: The welding robot moves, driving the welding head to perform welding according to the welding path; Step 4: When the end point of the weld is identified, or the welding timeout is exceeded, or the cumulative movement distance exceeds the set value, stop welding and move the welding robot back to the origin. The training steps for the deep learning model are as follows: Step a: Collect weld sample images and create a dataset; Step b: Modify the structure of the weld detection network CenterNet and the weld tracking network CenterTrack to obtain the weld feature extraction network; Step c: Perform self-supervised pre-training on the weld feature extraction network to obtain pre-trained weights; Step d: Train the modified weld detection network CenterNet and weld tracking network CenterTrack to obtain a deep learning model; Step (a) includes the following steps: Step (a1): Sample various weld joint structure workpieces to obtain a dataset; Step (a2): Label the weld joint structure type and welding feature point coordinates of the dataset in step (a1); Step (a3): Remove a portion of the workpieces from the batch of three-dimensional complex structure workpieces that need to be welded; use a teaching method to enable the robot to sample the weld along the welding path; Step (a4): Label the coordinates of welding feature points in the dataset obtained in step (a3), and select the images of the start and end points of the weld in the dataset as special types; Step (b) of modifying the structure of the weld detection network CenterNet and the weld tracking network CenterTrack specifically includes the following steps: Step (b1): Remove the size prediction branch of CenterNet, and modify the first loss function of the weld detection network structure as follows: ; The loss function for heatmap prediction, The loss function for predicting the offset of the heatmap center point is... These are coefficients used to control the weights between the loss functions; Step (b2): Remove the displacement prediction branch of CenterTrack, and modify the second loss function of the weld seam tracking network structure as follows: ; The loss function for heatmap prediction, The loss function for predicting the offset of the heatmap center point is... These are coefficients used to control the weights between the loss functions; Step (b3): ​​Modify the step of extracting peaks from the heatmap in CenterNet and CenterTrack, and directly output the maximum value of the heatmap as the identified weld location. Step (b4): Use the single-pass network structure search method to obtain CenterNet and CenterTrack feature extraction networks suitable for weld feature extraction; Step (b4) uses a single-pass network structure search method to obtain the CenterNet and CenterTrack feature extraction networks suitable for weld feature extraction, specifically including the following steps: Step (b41): Design the network search space and use the moving-flipping bottleneck convolution as the basic module of the network. The entire search space includes 4 different moving-flipping bottleneck convolution modules. Step (b42): Based on the designed network search space, train its hypergraph using the single-path uniform sampling method and the SGD gradient descent method on the weld seam image dataset until convergence; Step (b43): Develop evaluation criteria for the network, use a genetic algorithm to evaluate each possible feature extraction network using the parameters trained in the hypergraph, and search for the optimal network structure as the weld feature extraction network. Step (c) of performing self-supervised pre-training on the weld feature extraction network to obtain pre-training weights specifically includes the following steps: Step (c1): Use the weld feature extraction network obtained in step (b) as an encoder, and add a decoder composed of stacked deconvolution layers at the end of the network to form an encoder-decoder network structure. Step (c2): Input the masked weld seam image into the model and use the unmasked image as a supervision signal for training; Step (d) of training the modified weld detection network CenterNet and weld tracking network CenterTrack specifically includes the following steps: Step (d1): Initialize the modified weld detection network CenterNet using pre-trained weights; Step (d2): The dataset is augmented by random horizontal rotation, addition of random noise, and random translation to obtain the augmented data; Step (d3): Input the enhanced data into the modified weld inspection network CenterNet for iterative training; Step (d4): Initialize the weld tracking network CenterTrack using the trained weld detection network CenterNet model parameters; Step (d5): Combine the time-related images in the dataset into real video data, and at the same time, simulate the changes in the weld seam image during the sampling process by randomly scaling and translating some images, thereby generating a simulated video; Step (d6): Input the real video and the simulated video into the CenterTrack weld tracking network model for training.

2. The deep learning-based three-dimensional weld seam tracking method according to claim 1, characterized in that: Step 1 includes the following steps: Step (11): Input the first image. First, the weld tracking network CenterNet identifies the type of weld and the coordinates of the welding position in the first image. At the same time, a confidence score is output for each type. If the confidence score exceeds the threshold, it is considered a successful identification; otherwise, it is considered a failure. Step (12): Input the second image. Based on the three pieces of information—the first image, the second image, and the heatmap of the first image—the weld seam tracking network CenterTrack tracks and obtains the coordinates of the weld seam feature points in the second image and the confidence level corresponding to those coordinates. If the confidence level exceeds the threshold, the tracking of this frame is considered successful; otherwise, it is considered a failure. Here, the heatmap of the first image refers to the image obtained by convolving the output of the feature extraction network by the weld seam tracking network CenterNet. Step (13): If a tracking failure occurs during the tracking process, rerun CenterNet to detect and locate the new weld feature point coordinates.

3. A deep learning-based three-dimensional weld seam tracking device, applied in a three-dimensional weld seam tracking equipment, comprising a welding robot, a robot control cabinet, a line structured light sensor, a welding machine, a welding head, and an industrial control computer. The line structured light sensor and the welding head are coaxially mounted on the end effector of the welding robot via a mounting bracket. During tracking welding, the welding robot drives the end effector to move, and the welding head and the line structured light sensor simultaneously sample and weld the workpiece placed on the worktable; characterized in that: The three-dimensional weld seam tracking device includes: The first module is used to input real-time sampled images from the line structured light sensor into the trained deep learning model to obtain information on the weld joint structure type and weld feature points. The second module is used to convert the spatial transformation relationship obtained by calibration of weld feature point information into control points that guide the robot's movement to form a welding path, and then send it to the welding robot. The third module is used to enable the welding robot to drive the welding head to perform welding according to the welding path. The fourth module is used to stop welding and move the welding robot back to its origin when the end point of the weld is detected, the welding timeout is exceeded, or the cumulative movement distance exceeds the set value. The training steps for the deep learning model are as follows: Step a: Collect weld sample images and create a dataset; Step b: Modify the structure of the weld detection network CenterNet and the weld tracking network CenterTrack; Step c: Perform self-supervised pre-training on the weld feature extraction network to obtain pre-trained weights; Step d: Train the modified weld detection network CenterNet and weld tracking network CenterTrack to obtain a deep learning model; The three-dimensional weld seam tracking device performs the method as described in any one of claims 1 or 2.

4. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that is executed by a processor to implement the method as described in any one of claims 1 or 2.

Images