Welding seam tracking method, electronic equipment, storage medium and program product

Abstract

The invention discloses a welding seam tracking method, electronic equipment, a storage medium and a program product, relates to the technical field of welding, and aims to provide a welding seam tracking scheme based on a machine learning model. The method comprises the following steps: acquiring a welding image when welding equipment performs welding; based on the welding image and a welding seam detection model, determining a welding seam position under a camera coordinate system; wherein the welding seam detection model is used for determining a welding seam position according to a welding image; converting the welding seam position under the camera coordinate system to the welding seam position under the world coordinate system; and adjusting the position of the welding equipment based on the welding seam position under the world coordinates.

Description

Technical Field

[0001] The present application relates to the field of welding technology, and in particular to a weld tracking method, electronic equipment, storage medium, and program product. Background Art

[0002] As a method for permanently joining materials, welding plays a vital role in modern manufacturing. It is widely used in numerous fields, including oil and gas pipelines, construction engineering, automotive manufacturing, shipbuilding, and aerospace, and is a key process for constructing various metal structures and products.

[0003] As the manufacturing industry continues to move towards automation and intelligence, the automation and intelligent development of welding technology has become an inevitable trend.

[0004] In the process of welding automation, weld seam tracking technology is one of the core technologies to achieve high-quality welding, so how to achieve weld seam tracking has become our need. Summary of the Invention

[0005] The present application provides a weld tracking method, electronic device, storage medium and program product, which can predict the weld position based on the welding image through a machine learning model, and provide a weld tracking solution.

[0006] In a first aspect, the present application provides a weld tracking method, which includes: obtaining a welding image of a welding device during welding; determining the weld position in a camera coordinate system based on the welding image and a weld detection model; wherein the weld detection model is used to determine the weld position based on the welding image; converting the weld position in the camera coordinate system to the weld position in the world coordinate system; and adjusting the position of the welding device based on the weld position in the world coordinate system.

[0007] Optionally, the weld detection model includes an input layer, a first convolution block, a second convolution block, a third convolution block, a fourth convolution block, a fifth convolution block, and a fully connected layer in sequence; wherein, the first convolution block includes two convolution layers and one pooling layer; the second convolution block includes two convolution layers and one pooling layer; the third convolution block includes three convolution layers and one pooling layer; the fourth convolution block includes three convolution layers and one pooling layer; and the fifth convolution block includes three convolution layers and one pooling layer.

[0008] Optionally, a convolution block attention module is further included between the convolution layer and the pooling layer in each of the first convolution block, the second convolution block, the third convolution block, the fourth convolution block, and the fifth convolution block.

[0009] Optionally, the size of the convolution kernel of the convolution layer in the first M convolution blocks of the first convolution block, the second convolution block, the third convolution block, the fourth convolution block, and the fifth convolution block is a first size, and the size of the convolution kernel of the convolution layer in the last N convolution blocks is a second size; the first size is smaller than the second size; M and N are both positive integers less than 5, and the sum of M and N is 5.

[0010] Optionally, the method further includes: after obtaining a welding image of the welding equipment during welding, performing at least one of the following preprocessing on the welding image: removing salt and pepper noise and Gaussian noise in the welding image using median filtering and Gaussian filtering methods; enhancing the contrast of the welding image through histogram equalization technology; and extracting an image of a region of interest from the welding image using a region of interest algorithm.

[0011] Optionally, the method also includes: obtaining a training sample set; the training sample set includes multiple training samples; each training sample includes a welding image when the welding equipment is performing welding and a weld position label corresponding to the welding image; based on the training sample set, the initial model of the weld detection model is trained to obtain a weld detection model.

[0012] Optionally, each training sample in the training sample set also includes a weld type label corresponding to the welding image; based on the training sample set, the initial model of the weld detection model is trained to obtain the weld detection model, including: inputting one or more training samples into the initial model each time to obtain the predicted weld position label and the predicted weld type label predicted by the initial model; determining a first loss function based on the predicted weld position label and the weld position label in the training sample; determining a second loss function based on the predicted weld type label and the weld type label in the training sample; performing a weighted summation of the first loss function and the second loss function to obtain the loss function of the initial model; adjusting the parameters in the initial model based on the loss function until the iteration stop condition is reached to obtain the weld detection model; wherein the iteration stop condition includes: the number of times the training sample is input into the initial model reaches a number threshold, and / or the initial model prediction error is less than or equal to the error threshold.

[0013] The weld tracking device provided in an embodiment of the present application can capture a welding image of a welding device during welding; determine the weld position in a camera coordinate system based on the welding image and a weld detection model; wherein the weld detection model is used to determine the weld position based on the welding image; convert the weld position in the camera coordinate system to the weld position in world coordinates; and adjust the position of the welding device based on the weld position in world coordinates. A weld tracking solution is provided by using a machine learning model to predict the weld position based on the welding image.

[0014] In a second aspect, the present application provides a weld tracking device, which includes various functional modules used in the method described in the first aspect above.

[0015] In a third aspect, the present application provides an electronic device comprising: a processor and a memory; the memory stores instructions executable by the processor, and when the processor is configured to execute the aforementioned instructions, the electronic device implements the method described in the first aspect.

[0016] In a fourth aspect, the present application provides a readable storage medium, comprising: software instructions; when the software instructions are executed in an electronic device, the electronic device implements the method described in the first aspect.

[0017] In a fifth aspect, the present application provides a computer program product, comprising: computer instructions; when the computer instructions are executed in an electronic device, the electronic device implements the method described in the first aspect.

[0018] The beneficial effects of the second to fifth aspects above can be referred to those described in the first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 A schematic diagram of the composition of a weld tracking system provided in an embodiment of the present application;

[0021] Figure 2 A schematic flow chart of a weld tracking method provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of the histogram equalization effect provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of the VGG model structure provided in the embodiments of this application;

[0024] Figure 5 Schematic diagram of the VGG model network layered structure provided in the embodiment of the present application;

[0025] Figure 6 A schematic diagram of coordinate conversion provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of the composition of a weld tracking device provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of the composition of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0029] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature specified as "first" or "second" may explicitly or implicitly include one or more of such features. Throughout this application, unless otherwise specified, "plurality" means two or more.

[0030] In the embodiments of the present application, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, article, or device comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of other identical elements in the process, article, or device comprising the element.

[0031] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0032] As a method for permanently joining materials, welding plays a vital role in modern manufacturing. It is widely used in numerous fields, including oil and gas pipelines, construction engineering, automotive manufacturing, shipbuilding, and aerospace, and is a key process for constructing various metal structures and products.

[0033] As the manufacturing industry continues to move toward automation and intelligentization, the automation and intelligent development of welding technology is becoming an inevitable trend. Automated welding can improve production efficiency, ensure consistent welding quality, reduce labor intensity, and adapt to the needs of large-scale production. Intelligent welding further empowers welding systems with sensing, decision-making, and adaptive tracking capabilities, enabling them to handle complex and changing welding tasks and working conditions.

[0034] In the process of welding automation, weld seam tracking technology is one of the core technologies to achieve high-quality welding, so how to achieve weld seam tracking has become our need.

[0035] Based on this, the embodiments of the present application provide a weld tracking method, electronic device, storage medium and program product, which can determine the weld position based on the welding image through a machine learning model, thereby realizing automatic tracking of the weld.

[0036] The following is an introduction with reference to the accompanying drawings.

[0037] Figure 1 This is a schematic diagram of the composition of the weld tracking system provided in the embodiment of the present application. Figure 1 As shown, the weld seam tracking system includes: a molten pool vision camera 100 and a weld seam tracking device 200. The molten pool vision camera 100 and the weld seam tracking device 200 are communicatively connected.

[0038] The molten pool vision camera 100 can be integrated with the welding equipment, or can be provided separately from the welding equipment, which is not limited in the present embodiment.

[0039] The molten pool vision camera 100 can be used to collect welding images when the welding equipment is performing welding, and send the collected welding images to the weld tracking device 200.

[0040] For example, Figure 1 As shown, the molten pool vision camera may include a camera lens 10 and a power supply and data transmission interface 20. The camera lens 10 may be used to capture welding images when the welding equipment is welding, and these welding images may be sent to the weld tracking device 200 via the power supply and data transmission interface 20.

[0041] In some embodiments, before using the molten pool vision camera 100 to capture welding images, the camera parameters (such as focal length, aperture, and frame rate, etc.) can be set according to the welding scene and accuracy requirements to ensure that the weld feature information can be clearly captured while avoiding overexposure or underexposure.

[0042] The weld tracking device 200 may be an electronic device with computing and processing capabilities, such as a computer or a server.

[0043] Among them, the server can be a single server, or it can be a server cluster composed of multiple servers. In some implementations, the server cluster can also be a distributed cluster. Optionally, the server can also be implemented on a cloud platform. For example, the cloud platform can include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an inter-cloud, and a multi-cloud, etc., or any combination thereof. The embodiments of the present application are not limited to this.

[0044] The weld tracking device 200 can be used to determine the weld position based on the weld image captured by the weld pool vision camera 100, thereby performing weld tracking. The specific process can be referred to the weld tracking method provided in the following method embodiment, and will not be repeated here.

[0045] It should be noted that the above description uses the example of the melt pool vision camera 100 and the weld seam tracking device 200 as independent devices. Optionally, the melt pool vision camera 100 and the weld seam tracking device 200 can also be integrated into one device. That is, the melt pool vision camera 100 or its corresponding functions and the weld seam tracking device 200 or its corresponding functions can be integrated into a single device. For example, a melt pool vision camera with a weld seam tracking function, or welding equipment with image acquisition and weld seam tracking functions, etc. This embodiment of the present application is not limited to this.

[0046] The execution subject of the weld tracking method provided in the embodiment of the present application is a weld tracking device (such as the weld tracking device 200 mentioned above). As mentioned above, the weld tracking device can be an electronic device with computing and processing functions such as a computer or a server. Optionally, the weld tracking device can also be a processor (such as a central processing unit (CPU)) in the aforementioned electronic device; or, the weld tracking device can also be an application (application, APP) with a weld tracking function installed in the aforementioned electronic device; or, the weld tracking device can also be a platform or software system deployed in the aforementioned electronic device; or, the weld tracking device can also be a functional module in the aforementioned electronic device for executing the weld tracking method. The embodiment of the present application does not limit this.

[0047] For the sake of simplicity, the following description will be made by taking the weld tracking method provided in the embodiments of the present application as an example in which the weld tracking device is the executing entity.

[0048] Figure 2 Schematic diagram of the flow of the weld tracking method provided in the embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0049] S101: Acquire a welding image of a welding device during welding.

[0050] As an example, as described above, the weld seam tracking device is in communication with the molten pool vision camera. The molten pool vision camera can be used to capture welding images of the welding equipment during welding and transmit the welding images to the weld seam tracking device. In this case, the weld seam tracking device can specifically receive the welding images transmitted from the molten pool vision camera.

[0051] In some embodiments, after S101 above, the weld tracking device may further perform at least one of the following preprocessing on the acquired welding image:

[0052] 1) Image denoising: Median filtering and Gaussian filtering are used to remove salt and pepper noise and Gaussian noise in the welding image while retaining the weld edge information.

[0053] 2) Image contrast enhancement: The contrast of the welding image is enhanced by using histogram equalization technology.

[0054] 3) Region of Interest (ROI) extraction: The image of the region of interest is extracted from the welding image using a region of interest algorithm. For example, the ROI is determined based on the approximate position and shape of the weld, thereby reducing the amount of subsequent calculations and improving the algorithm processing speed.

[0055] For example, Figure 3 This is a schematic diagram of the histogram equalization effect provided in the embodiment of the present application. Figure 3 As shown in the figure, before histogram equalization, the number of pixels (vertical axis) corresponding to different grayscales (horizontal axis) in the original histogram is relatively unevenly distributed. After equalization, the distribution gap of the number of pixels corresponding to different grayscales in the histogram is narrowed, and the distribution is more balanced.

[0056] S102: Determine the weld position in the camera coordinate system based on the welding image and the weld detection model.

[0057] The weld detection model is used to determine the weld position based on the weld image. The specific training process of the weld detection model can be referred to the following embodiment and will not be repeated here.

[0058] In some embodiments, the weld detection model may specifically adopt a Visual Geometry Group (VGG) model.

[0059] For example, Figure 4 This is a schematic diagram of the VGG model structure provided in the embodiment of this application. Figure 4 As shown, the model includes: input layer ( Figure 4Input is shown as an example), the first convolutional layer (the convolution kernel size of this convolutional layer is 3×3 and the number of channels is 64, so Figure 4 3×3conv, 64 is shown as an example), the second convolutional layer (the convolution kernel size of this convolutional layer is 3×3 and the number of channels is 64, so Figure 4 3×3conv, 64 is shown as an example), the first pooling layer ( Figure 4 Pool is used as an example), the third convolutional layer (the convolution kernel size of this convolutional layer is 3×3 and the number of channels is 128, so Figure 4 3×3conv, 128 is used as an example), the fourth convolutional layer (the convolution kernel size of this convolutional layer is 3×3 and the number of channels is 128, so Figure 4 3×3conv, 128 is used as an example), the second pooling layer ( Figure 4 Pool is used as an example), the fifth convolutional layer (the convolution kernel size of this convolutional layer is 3×3 and the number of channels is 256, so Figure 4 3×3conv,64 is shown as an example), the sixth convolutional layer (the convolution kernel size of this convolutional layer is 3×3 and the number of channels is 256, so Figure 4 3×3conv, 64 is shown as an example), the seventh convolutional layer (the convolution kernel size of this convolutional layer is 3×3 and the number of channels is 256, so Figure 4 3×3conv, 64 is shown as an example), the third pooling layer ( Figure 4 Pool is used as an example), the eighth convolutional layer (the convolution kernel size of this convolutional layer is 3×3 and the number of channels is 512, so Figure 4 3×3conv, 64 is shown as an example), the ninth convolutional layer (the convolution kernel size of this convolutional layer is 3×3 and the number of channels is 512, so Figure 4 3×3conv, 64 is shown as an example), the tenth convolutional layer (the convolution kernel size of this convolutional layer is 3×3 and the number of channels is 512, so Figure 4 3×3conv, 64 is shown as an example), the fourth pooling layer ( Figure 4 Pool is used as an example), the eleventh convolutional layer (the convolution kernel size of this convolutional layer is 3×3 and the number of channels is 512, so Figure 4 3×3conv, 64 is shown as an example), the twelfth convolution layer (the convolution kernel size of this convolution layer is 3×3 and the number of channels is 512, so Figure 4 3×3conv, 64 is shown as an example), the thirteenth convolutional layer (the convolution kernel size of this convolutional layer is 3×3 and the number of channels is 512, so Figure 4 3×3conv, 64 is shown as an example), the fifth pooling layer ( Figure 4Pool is shown as an example), and three fully connected layers ( Figure 4 FC 4096, FC 4096, FC 100, and Softmax are used as examples.

[0060] In some embodiments, a combination of consecutive convolutional layers and pooling layers can also be viewed as a convolutional block. In this case, Figure 5 This is a schematic diagram of the VGG model network layered structure provided in the embodiment of this application. Figure 5 As shown above Figure 4 The first convolutional layer in ( Figure 5 Conv1_1 is used as an example), the second convolutional layer ( Figure 5 Conv1_2 is used as an example), and the first pooling layer ( Figure 5 Pool 1 is used as an example in the figure) and can be regarded as a first convolution block. Figure 4 The third convolutional layer in ( Figure 5 Conv2_1 is used as an example), the fourth convolutional layer ( Figure 5 Conv2_2 is used as an example), and the second pooling layer ( Figure 5 Pool 2 is used as an example in the figure) and can be regarded as a second convolution block. Figure 4 The fifth convolutional layer in ( Figure 5 Conv3_1 is used as an example), the sixth convolutional layer ( Figure 5 Conv3_2 is used as an example), the seventh convolutional layer ( Figure 5 Conv3_3 is used as an example), and the third pooling layer ( Figure 5 Pool 3 is used as an example in the figure) and can be regarded as a third convolution block. Figure 4 The eighth convolutional layer in ( Figure 5 Conv4_1 is used as an example), the ninth convolutional layer ( Figure 5 Conv4_2 is used as an example), the tenth convolutional layer ( Figure 5 Conv4_3 is used as an example), and the fourth pooling layer ( Figure 5 Pool 4 is used as an example in the figure) and can be regarded as a fourth convolution block. Figure 4 The eleventh convolutional layer in ( Figure 5 Conv5_1 is used as an example), the twelfth convolutional layer ( Figure 5 Conv5_2 is used as an example), the thirteenth convolutional layer ( Figure 5 Conv5_3 is used as an example), and the fifth pooling layer ( Figure 5Pool 5 is used as an example in the figure and can be regarded as the fifth convolutional block. The three fully connected layers after the five convolutional blocks are represented as FC 6, FC 7, and FC 8 respectively.

[0061] In some embodiments, the weld tracking device may further introduce a convolutional block attention module (CBAM) into the network of the VGG model, so that the network can automatically focus on a given key area (i.e., the weld area) in the welding image, suppress the interference of irrelevant background information, and further improve the accuracy and effectiveness of feature extraction. In this case, a convolutional block attention module may also be included between the convolution layer and the pooling layer in each of the first convolution block, the second convolution block, the third convolution block, the fourth convolution block, and the fifth convolution block.

[0062] In some embodiments, the weld tracking device can also use smaller convolution kernels in the shallow network to extract detailed features and larger convolution kernels in the deep network to focus on overall features based on the original network. In this case, it can also be understood that the size of the convolution kernel of the convolution layer in the first M convolution blocks of the first convolution block, the second convolution block, the third convolution block, the fourth convolution block, and the fifth convolution block is the first size, and the size of the convolution kernel of the convolution layer in the last N convolution blocks is the second size.

[0063] The first size is smaller than the second size, M and N are both positive integers smaller than 5, and the sum of M and N is 5.

[0064] As an example, the first size may be Figure 3 The second size is 3×3 as shown in FIG, and may be 5×5 or 7×7, etc. The embodiment of the present application does not limit the specific values of the first size and the second size.

[0065] As an example, the convolution kernels of the convolution layers in the first two convolution blocks of the above five convolution blocks can adopt the first size, and the sizes of the convolution kernels of the convolution layers in the last three convolution blocks can adopt the second size, that is, M can be 2 and N can be 3.

[0066] In some embodiments, the above-mentioned pooling layer can be used to reduce the aggregation dimension. The embodiment of the present application can improve the maximum pooling of the original network into adaptive pooling. The adaptive pooling can automatically determine the pooling window size and step size according to the given output size, so that when the model faces input images of different sizes, the output feature map size is fixed, thereby improving the versatility and stability of subsequent processing of different input image sizes.

[0067] In some embodiments, each convolutional layer in the above-mentioned VGG model network may further include an activation function layer. In the embodiment of the present application, the activation function layer may be replaced with an improved version of the ReLU function, Leaky ReLU. Leaky ReLU can assign a smaller non-zero slope when the input is less than 0, so that the negative half-axis also has a certain gradient information transmission, alleviating the neuron "death" problem that may occur in ReLU, and helping to improve the stability and convergence speed of model training.

[0068] S103: Convert the weld position in the camera coordinate system to the weld position in the world coordinate system.

[0069] For example, Figure 6 This is a schematic diagram of coordinate conversion provided by the embodiment of the present application. Figure 6 As shown, the weld tracking device can convert the coordinates in the camera coordinate system 12 into coordinates in the world coordinate system 22 by means of rotation+translation.

[0070] For example, the conversion relationship between the coordinates in the camera coordinate system 12 and the coordinates in the world coordinate system 22 can be specifically shown as the following formula (1):

[0071]

[0072] In formula (1), X c 、Y c , and Z c Respectively represent the X-axis coordinate, Y-axis coordinate, and Z-axis coordinate in the camera coordinate system. w 、Y w , and Z w Respectively represent the X-axis coordinate, Y-axis coordinate, and Z-axis coordinate in the world coordinate system. 00 、R 01 , and R 02 Represents the direction vector of the world coordinate system X axis in the camera coordinate system. 10 、R 11 , and R 12 Represents the direction vector of the Y axis of the world coordinate system in the camera coordinate system. 20 、R 21 , and R 22 Represents the direction vector of the Z axis of the world coordinate system in the camera coordinate system. x Indicates the offset of the world coordinate system point along the X axis of the camera coordinate system, T y Indicates the offset of the world coordinate system point along the Y axis of the camera coordinate system, T z Indicates the offset of the world coordinate system point along the Z axis of the camera coordinate system.

[0073] S104: Adjust the position of the welding equipment based on the weld position in the world coordinates.

[0074] As an example, the weld tracking device can compare the weld position in the world coordinates with the current position of the welding equipment, calculate the position deviation and angle deviation, and then move the current position of the welding equipment according to the position deviation and angle deviation to adjust the position of the welding equipment.

[0075] The weld tracking device provided in an embodiment of the present application can capture a welding image of a welding device during welding; determine the weld position in a camera coordinate system based on the welding image and a weld detection model; wherein the weld detection model is used to determine the weld position based on the welding image; convert the weld position in the camera coordinate system to the weld position in world coordinates; and adjust the position of the welding device based on the weld position in world coordinates. A weld tracking solution is provided by using a machine learning model to predict the weld position based on the welding image.

[0076] In some possible embodiments, before the above S101, the weld tracking device may also obtain a trained or pre-trained weld detection model.

[0077] In one possible implementation, the weld tracking device may directly obtain a trained or pre-trained weld detection model from other devices.

[0078] In another possible implementation, the weld tracking device may also train a weld detection model based on training samples. In this case, before the above S101, the method may further include the following steps:

[0079] Step 1a: Obtain a training sample set.

[0080] The training sample set includes multiple training samples, and each training sample includes a welding image when the welding equipment is performing welding and a weld position label corresponding to the welding image.

[0081] As an example, the weld tracking device can collect a large number of welding images under different working conditions, including welding images of different materials, different sizes, different welding processes (such as manual arc welding and gas shielded welding) and different welding stages (such as arc starting, welding process, and arc ending). During the image acquisition process, ensure that the shooting angle and distance parameters of the molten pool vision camera are diverse to increase the richness of the data set. The collected images are then accurately annotated, and the annotation content may include the location, type, and feature point coordinate information of the weld. The annotation work can be assisted by a professional annotation tool (LabelMe). The annotated welding images can be divided into training sample sets, verification sample sets, and test sample sets, and the ratio can be based on actual conditions (for example, it can be set to 7:2:1 or 8:1:1, etc.).

[0082] As an example, the welding images in multiple training samples may include welding images of different sizes. By designing welding images of different sizes, feature information of different scales can be expressed, so that the model can focus on both the detailed features and the overall features of the welds, thereby improving the recognition ability of different types of welds and welds under different working conditions.

[0083] It can be understood that training using weld position labels can be understood as a regression task, which is mainly used to determine the precise position of weld feature points (such as the weld starting point, inflection point, and end point, etc.) in the welding image, and output the coordinate information of the feature points through the model to achieve precise positioning of the weld.

[0084] Optionally, for applications that require weld type determination (welds from different connection methods), a classification task can be added. The weld detection model can be used to classify weld types, classifying weld images into corresponding categories to provide a basis for subsequent welding parameter adjustments. In other words, the training samples can also include weld type labels corresponding to the weld images.

[0085] Step 2a: Based on the training sample set, the initial model of the weld detection model is trained to obtain the weld detection model.

[0086] In one possible implementation, as described above, the training samples may include weld position labels and weld type labels. In this case, step 2a may specifically include the following steps:

[0087] Step 2.1a: Input one or more training samples into the initial model each time to obtain the predicted weld position label and the predicted weld type label predicted by the initial model.

[0088] Step 2.2a: Determine a first loss function based on the predicted weld position labels and the weld position labels in the training samples.

[0089] As an example, the first loss function may adopt a mean square error loss function.

[0090] In some embodiments, in order to solve the problem of unbalanced training samples (for example, the number of pixels in the weld area is relatively small compared to the background), Online Hard Example Mining (OHEM) and DiceLoss strategies can be introduced to make the model pay more attention to difficult-to-classify samples and improve its learning ability for the weld area.

[0091] Step 2.3a: Determine a second loss function based on the predicted weld type label and the weld type label in the training sample.

[0092] As an example, the second loss function may adopt a cross entropy loss function.

[0093] Step 2.4a: Perform weighted summation on the first loss function and the second loss function to obtain the loss function of the initial model.

[0094] In some embodiments, the weld tracking device can also use Dropout regularization to prevent model overfitting and improve the model's generalization ability. During the training process, by adding a regularization term to the loss function, the model's complexity is penalized, making the features learned by the model on the training samples more generalizable.

[0095] Step 2.5a: Adjust the parameters in the initial model based on the loss function until the iteration stop condition is reached to obtain the weld detection model.

[0096] The iteration stopping conditions include: the number of times the training sample is input into the initial model reaches a threshold number, and / or the initial model prediction error is less than or equal to an error threshold.

[0097] In some possible embodiments, the Adam optimization algorithm can be used during training to dynamically adjust the model's hyperparameters, ensuring gradual convergence of the loss function. Furthermore, appropriate training epochs and batch size parameters are set to ensure the model fully learns the features of the training samples, avoiding overfitting or regression. Furthermore, the regularization coefficient is adjusted based on the model's performance on the validation set to balance the model's complexity and fitting ability.

[0098] In the weld tracking method provided in the present application, the weld detection model is trained with a large amount of labeled data, enabling it to grasp weld characteristic patterns under complex working conditions, accurately identify weld location and shape, and reduce tracking errors. The use of multi-scale feature fusion and an attention mechanism enables the algorithm to take into account both macroscopic and microscopic weld characteristics, avoiding errors caused by local information issues, enhancing tracking accuracy, improving the first-pass weld pass rate, and avoiding welding defects caused by weld deviations (such as undercuts and incomplete penetration).

[0099] Furthermore, the improved VGG network in this application is highly robust to noise. During the welding process, despite strong noise interference from arc light, spatter, smoke, and other sources, the algorithm can effectively filter out noise and accurately extract weld features by learning from various noise scenarios during training. By learning from weld images of varying shapes and states, the algorithm can automatically adjust its weld feature recognition strategy. This allows it to accurately track welds even when the welded workpiece exhibits a certain degree of deformation or machining errors, improving the welding system's adaptability to complex working conditions.

[0100] The improved VGG network in this application has a simple and efficient architecture. During the training process, the real-time requirements of actual welding scenarios are fully considered. By optimizing the model parameters and training strategies, the trained model has high operating efficiency while ensuring accuracy. In actual applications, it can complete the processing of a frame of image within the specified time, achieve stable real-time weld tracking without manual intervention, greatly shorten the welding cycle, and improve production efficiency.

[0101] Finally, the improved VGG in this application can be easily integrated with other intelligent welding technologies (intelligent welding parameter control, welding robot path planning, etc.), further improving the intelligence level and overall performance of the welding system, and providing good scalability for the future development of welding technology.

[0102] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to realize the above functions, the weld tracking device includes hardware structures and / or software modules corresponding to the execution of each function. It should be easy to realize that the technical goals in this field are combined with the algorithm steps of each example described in the embodiments disclosed in this article, and the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technical goals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0103] In an exemplary embodiment, the present application provides a weld tracking device in the form of a virtual device. Figure 7This is a schematic diagram of the composition of the weld tracking device provided in the embodiment of the present application. Figure 7 As shown, the weld tracking device includes: an acquisition module 701 and a processing module 702.

[0104] The acquisition module 701 is used to acquire welding images when the welding equipment is performing welding.

[0105] Processing module 702 is used to determine the weld position in the camera coordinate system based on the welding image and the weld detection model; wherein the weld detection model is used to determine the weld position based on the welding image; convert the weld position in the camera coordinate system to the weld position in the world coordinate system; and adjust the position of the welding equipment based on the weld position in the world coordinate system.

[0106] In some possible embodiments, the weld detection model includes, in sequence, an input layer, a first convolution block, a second convolution block, a third convolution block, a fourth convolution block, a fifth convolution block, and a fully connected layer; wherein, the first convolution block includes two convolution layers and one pooling layer; the second convolution block includes two convolution layers and one pooling layer; the third convolution block includes three convolution layers and one pooling layer; the fourth convolution block includes three convolution layers and one pooling layer; and the fifth convolution block includes three convolution layers and one pooling layer.

[0107] In some possible embodiments, a convolution block attention module is further included between the convolution layer and the pooling layer in each of the first convolution block, the second convolution block, the third convolution block, the fourth convolution block, and the fifth convolution block.

[0108] In some possible embodiments, the size of the convolution kernel of the convolution layer in the first M convolution blocks of the first convolution block, the second convolution block, the third convolution block, the fourth convolution block, and the fifth convolution block is a first size, and the size of the convolution kernel of the convolution layer in the last N convolution blocks is a second size; the first size is smaller than the second size; M and N are both positive integers less than 5, and the sum of M and N is 5.

[0109] In some possible embodiments, the processing module 702 is further used to perform at least one of the following preprocessing on the welding image after the acquisition module 701 acquires the welding image of the welding equipment during welding: using median filtering and Gaussian filtering methods to remove salt and pepper noise and Gaussian noise in the welding image; enhancing the contrast of the welding image through histogram equalization technology; and extracting an image of a region of interest from the welding image using a region of interest algorithm.

[0110] In some possible embodiments, the acquisition module 701 is also used to obtain a training sample set; the training sample set includes multiple training samples; each training sample includes a welding image of the welding equipment during welding and a weld position label corresponding to the welding image; the processing module 702 is also used to train the initial model of the weld detection model based on the training sample set to obtain a weld detection model.

[0111] In some possible embodiments, each training sample in the training sample set also includes a weld type label corresponding to the welding image; the processing module 702 is specifically used to input one or more training samples into the initial model each time to obtain a predicted weld position label and a predicted weld type label predicted by the initial model; based on the predicted weld position label and the weld position label in the training sample, a first loss function is determined; based on the predicted weld type label and the weld type label in the training sample, a second loss function is determined; a weighted sum of the first loss function and the second loss function is performed to obtain the loss function of the initial model; based on the loss function, the parameters in the initial model are adjusted until the iteration stop condition is reached to obtain a weld detection model; wherein the iteration stop condition includes: the number of times the training sample is input into the initial model reaches a number threshold, and / or the initial model prediction error is less than or equal to the error threshold.

[0112] In an exemplary embodiment, as described above, the weld tracking device may be an electronic device such as a computer or an electronic device having a computing and processing function. In this case, an embodiment of the present application further provides an electronic device. Figure 8 This is a schematic diagram of the composition of the electronic device provided in the embodiment of the present application. Figure 8 As shown, the electronic device includes:

[0113] Processor 802 , communication interface 803 , and bus 804 . As an example, the electronic device 80 may further include a memory 801 .

[0114] Processor 802 may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 802 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array, or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 802 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, or a combination of a DSP and a microprocessor.

[0115] The communication interface 803 is used to connect to other devices via a communication network, such as Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0116] The memory 801 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0117] As a possible implementation, memory 801 can exist independently of processor 802. Memory 801 can be connected to processor 802 via bus 804 to store instructions or program codes. When processor 802 calls and executes the instructions or program codes stored in memory 801, the weld tracking method provided in the embodiments of the present application can be implemented.

[0118] In another possible implementation, the memory 801 may also be integrated with the processor 802 .

[0119] The bus 804 may be an extended industry standard architecture (EISA) bus, etc. The bus 804 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0120] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the electronic device can be divided into different functional modules to complete all or part of the functions described above.

[0121] In an exemplary embodiment, the present application also provides a readable storage medium including software instructions, which, when executed in an electronic device, enables the electronic device to execute any one of the methods provided in the above embodiments.

[0122] In an exemplary embodiment, the present application also provides a computer program product, including computer instructions, which, when executed in an electronic device, enables the electronic device to execute any one of the methods provided in the above embodiments.

[0123] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using a software program, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer-executable instructions. When the computer-executable instructions are loaded and executed on a computer, the process or function according to the embodiments of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer-executable instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer-executable instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), or an optical medium (e.g., a DVD).

[0124] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple components. A single processor or other unit may implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0125] Although the present application has been described with reference to specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely illustrative of the present application as defined by the appended claims and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, the present application is intended to include such modifications and variations as fall within the scope of the claims of the present application and their equivalents.

[0126] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A weld tracking method, characterized in that: The method comprises: Acquire welding images of welding equipment during welding; Determining the weld position in a camera coordinate system based on the welding image and the weld detection model; wherein the weld detection model is used to determine the weld position based on the welding image; Converting the weld position in the camera coordinate system to the weld position in the world coordinate system; The position of the welding device is adjusted based on the weld seam position in world coordinates.

2. The method according to claim 1, characterized in that The weld detection model includes an input layer, a first convolution block, a second convolution block, a third convolution block, a fourth convolution block, a fifth convolution block, and a fully connected layer in sequence; Among them, the first convolution block includes two convolution layers and one pooling layer; the second convolution block includes two convolution layers and one pooling layer; the third convolution block includes three convolution layers and one pooling layer; the fourth convolution block includes three convolution layers and one pooling layer; the fifth convolution block includes three convolution layers and one pooling layer.

3. The method according to claim 2, characterized in that A convolution block attention module is further included between the convolution layer and the pooling layer in each of the first convolution block, the second convolution block, the third convolution block, the fourth convolution block, and the fifth convolution block.

4. The method according to claim 2, characterized in that The size of the convolution kernel of the convolution layer in the first M convolution blocks of the first convolution block, the second convolution block, the third convolution block, the fourth convolution block, and the fifth convolution block is a first size, and the size of the convolution kernel of the convolution layer in the last N convolution blocks is a second size; the first size is smaller than the second size; M and N are both positive integers less than 5, and the sum of M and N is 5.

5. The method according to claim 1, wherein The method further comprises: After obtaining the welding image of the welding device during welding, performing at least one of the following preprocessing on the welding image: Using median filtering and Gaussian filtering to remove salt and pepper noise and Gaussian noise in the welding image; enhancing the contrast of the welding image by using a histogram equalization technique; An image of a region of interest is extracted from the welding image using a region of interest algorithm.

6. The method according to claim 1, characterized in that The method further comprises: Obtaining a training sample set; the training sample set includes a plurality of training samples; each training sample includes a welding image of a welding device performing welding and a weld position label corresponding to the welding image; Based on the training sample set, the initial model of the weld detection model is trained to obtain the weld detection model.

7. The method according to claim 6, characterized in that Each training sample in the training sample set further includes a weld type label corresponding to the welding image; The initial model of the weld detection model is trained based on the training sample set to obtain the weld detection model, including: Inputting one or more training samples into the initial model each time to obtain a predicted weld position label and a predicted weld type label predicted by the initial model; Determining a first loss function based on the predicted weld position label and the weld position label in the training sample; Determining a second loss function based on the predicted weld type label and the weld type label in the training sample; Performing a weighted summation on the first loss function and the second loss function to obtain a loss function of the initial model; Adjusting the parameters in the initial model based on the loss function until an iteration stop condition is reached to obtain the weld detection model; The iteration stopping condition includes: the number of times the training sample is input into the initial model reaches a number threshold, and / or the prediction error of the initial model is less than or equal to an error threshold.

8. An electronic device, characterized in that: include: processor and memory; The memory stores instructions executable by the processor; When the processor is configured to execute the instructions, the electronic device implements the method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that: include: Software instructions; When the software instructions are executed in an electronic device, the electronic device is enabled to implement the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that include: Computer instructions; When the computer instructions are executed in an electronic device, the electronic device is enabled to implement the method according to any one of claims 1 to 7.

Images