Noise-resistant weld feature recognition method and system based on lightweight neural network

Abstract

The present invention relates to the field of weld recognition technology, and specifically discloses a noise-resistant weld feature recognition method and system based on a lightweight neural network, including: acquiring welding image data during the welding process; inputting the acquired welding image data into a trained weld feature recognition model, and outputting the type and position information of the weld; wherein the weld feature recognition model includes a backbone network, a neck layer, and a head layer; the backbone network outputs feature information of different scales, and transmits the feature information to the neck layer; the neck layer fuses feature information of different scales, and the fused feature map is input into the head layer; the head layer includes three branches, namely, a heat map branch, an offset branch, and a classification branch, and finally predicts the position and type of the weld. The present invention is used to accurately classify and locate weld feature points using single-line structured light vision, while reducing the parameters and calculation amount of the model, and meeting the real-time requirements of welding tracking systems in the industrial field.

Description

Technical Field

[0001] The present invention relates to the technical field of weld feature recognition and processing, and in particular to a noise-resistant weld feature recognition method and system based on a lightweight neural network. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the widespread application of robots in the field of industrial manufacturing, a welding method based on teaching and reproduction has emerged. Traditional robot welding relies on workers to manually set the welding gun position trajectory, which is inefficient and easily affected by the time-varying size error of the workpiece (such as thermal deformation). In order to track the weld in real time and realize the automatic adjustment of the welding robot, researchers have designed some advanced sensors to obtain the spatial position information and category information of the weld. With the rapid development of machine vision, structured light vision has been widely used in intelligent robots due to its high precision and the advantages of providing rich welding process feature information.

[0004] The features of single-line structured light projection fringes of different welds vary greatly. This difference is very valuable for obtaining position information related to weld feature points and facilitates the classification of fringes. This classification is essential for effectively adjusting welding process parameters. However, common noise in the welding process, including spatter, smoke, and arc reflection light, can be confused with the fringes collected by industrial cameras. Some morphological-based methods have been proposed to achieve weld feature extraction. There are two common problems in morphological-based research. First, feature point (or region) extraction and weld classification usually require two stages. On the other hand, although these models have certain noise resistance, their tracking accuracy will be reduced when the noise is continuously high.

[0005] With the development of deep learning technology in the field of computer vision, object detection and semantic segmentation processes have been improved and enhanced in the context of improving the hardware performance of computing devices and proposing convolutional neural network (CNN) models. These tasks are often transferred to the task of seam feature extraction using line structured light, which helps to create an anti-noise model for effective feature extraction. For example, the YOLO-WELD model was established based on YOLOv5 to detect welding feature points. There are also methods for extracting multi-layer and multi-pass weld feature points using conditional generative adversarial networks (CGAN) and improved CNN models.

[0006] However, these neural network-based methods require the allocation of a large amount of computing resources; this in turn requires the central control device integrated into the welding robot system to have higher computing performance. Industrial computers usually do not have independent GPU computing units and lack GPU acceleration support, so they cannot meet the real-time requirements of welding. Summary of the invention

[0007] In order to solve the above problems, the present invention proposes a noise-resistant weld feature recognition method and system based on a lightweight neural network. Based on the lightweight neural network design, while having strong anti-interference ability, it maintains a high inference frequency to meet the real-time requirements of the welding tracking system in the industrial field.

[0008] In some embodiments, the following technical solutions are adopted:

[0009] A noise-resistant weld feature recognition method based on a lightweight neural network, comprising:

[0010] Acquire welding image data during welding process;

[0011] Input the acquired welding image data into the trained weld feature recognition model, and output the type and location information of the weld;

[0012] Among them, the weld feature recognition model includes a backbone network, a neck layer and a head layer; the backbone network outputs feature information of different scales and transmits the feature information to the neck layer; the neck layer fuses the feature information of different scales, and the fused feature map is input into the head layer; the head layer includes three branches: a heat map branch, an offset branch and a classification branch. The heat map branch is used to output a heat map tensor to obtain the probability that each area contains a feature point; the offset branch is used to output an offset tensor to obtain the position deviation of the feature points generated in each dimension; the classification branch is used to output a classification tensor to obtain the corresponding score of each weld type; and finally the position and type of the weld are predicted.

[0013] In other embodiments, the following technical solutions are adopted:

[0014] A noise-resistant weld feature recognition system based on a lightweight neural network, comprising:

[0015] A data acquisition module, used for acquiring welding image data during the welding process;

[0016] The weld prediction module is used to input the acquired welding image data into the trained weld feature recognition model and output the type and location information of the weld;

[0017] Among them, the weld feature recognition model includes a backbone network, a neck layer and a head layer; the backbone network outputs feature information of different scales and transmits the feature information to the neck layer; the neck layer fuses the feature information of different scales, and the fused feature map is input into the head layer; the head layer includes three branches: a heat map branch, an offset branch and a classification branch. The heat map branch is used to output a heat map tensor to obtain the probability that each area contains a feature point; the offset branch is used to output an offset tensor to obtain the position deviation of the feature points generated in each dimension; the classification branch is used to output a classification tensor to obtain the corresponding score of each weld type; and finally the position and type of the weld are predicted.

[0018] In other embodiments, the following technical solutions are adopted:

[0019] A terminal device includes a processor and a memory, wherein the processor is used to implement instructions; the memory is used to store multiple instructions, and the instructions are suitable for being loaded by the processor and executing the above-mentioned noise-resistant weld feature recognition method based on lightweight neural network.

[0020] In other embodiments, the following technical solutions are adopted:

[0021] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor of a terminal device and executing the above-mentioned noise-resistant weld feature recognition method based on a lightweight neural network.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] (1) The present invention constructs a lightweight neural network structure based on MobileNetv3, which is used to accurately classify and locate weld feature points using single-line structured light vision, while reducing the parameters and calculation amount of the model, so that the model can be deployed on computing platforms that cannot obtain GPU acceleration, such as embedded industrial computers (EICs), while maintaining a high inference frequency to meet the real-time requirements of welding tracking systems in the industrial field.

[0024] (2) The neural network structure of the present invention uses DP (Depthwise Separable Convolution) blocks to replace standard convolution layers in the neck layer and the head layer. The DP block decomposes the standard convolution operation into depthwise convolution and point-by-point convolution, which significantly reduces the number of parameters and the amount of calculation of the model. Due to the reduction in parameters and the amount of calculation, the memory occupied by the model during runtime is also reduced accordingly. At the same time, the training and inference speed of the model is usually faster, and it is easier to adapt to different hardware platforms, so that better lightweight performance can be obtained, real-time requirements can be met, and deployment costs can be reduced.

[0025] (3) The neural network structure of the present invention adds a cascade channel attention module (CCAM) at the neck layer and uses CCAM to filter noise. At the same time, CCAM adopts a cascade group structure, which can reduce the amount of calculation without almost losing accuracy.

[0026] (4) The head layer of the present invention outputs a heat map tensor, an offset tensor and a classification tensor respectively. First, the model recognizes the first two partitions with the highest prediction scores in the heat map tensor, and then combines these data with the corresponding segments on the offset tensor to determine the final weld location prediction result. The heat map tensor is usually used to indicate the position of the weld in the image, and the offset tensor can provide a finer position adjustment. Combining these two types of information can more accurately locate the weld.

[0027] Other features and advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Schematic diagram of the weld feature recognition model structure in an embodiment of the present invention;

[0029] Figure 2 Schematic diagram of the DP block structure in an embodiment of the present invention;

[0030] FIG3 (a) is an original schematic diagram of a feature pyramid network structure in an embodiment of the present invention;

[0031] FIG3( b ) is a schematic diagram of an improved feature pyramid network structure in an embodiment of the present invention;

[0032] Figure 4 Schematic diagram of the structure of the cascade channel attention module in an embodiment of the present invention;

[0033] Figure 5 This is a diagram showing the effect of feature point recognition after optimizing the pooling filter in an embodiment of the present invention;

[0034] FIG6 (a) is a confusion matrix obtained by the model of the embodiment of the present invention on the adjusted test set;

[0035] Figure 6 (b) shows the confusion matrix obtained by the YOLOv5n model on the adjusted test set;

[0036] Figure 6 (c) shows the confusion matrix obtained by the DETR model on the adjusted test set;

[0037] Figure 7: are absolute value error curves of the weld feature recognition model in the embodiment of the present invention in the X and Y directions evaluated on the low noise and high noise test sets respectively; wherein, (a) is the absolute value error of the feature point regression in the X direction of the model of this embodiment evaluated on the low noise test set; (b) is the absolute value error of the feature point regression in the Y direction of the model of this embodiment evaluated on the low noise test set; (c) is the absolute value error of the feature point regression in the X direction of the model of this embodiment tested on the high noise test set; (d) is the absolute value error of the feature point regression in the Y direction of the model of this embodiment tested on the high noise test set;

[0038] Figure 8 The absolute error curves of the X and Y directions of the YOLOv5 model evaluated on the low-noise and high-noise test sets, respectively; where (a) is the absolute error of the feature point regression in the X direction of the YOLOv5 model evaluated on the low-noise test set; (b) is the absolute error of the feature point regression in the Y direction of the YOLOv5 model evaluated on the low-noise test set; (c) is the absolute error of the feature point regression in the X direction of the YOLOv5 model tested on the high-noise test set; (d) is the absolute error of the feature point regression in the Y direction of the YOLOv5 model tested on the high-noise test set;

[0039] Fig. 9 The absolute error curves of the DETR model in the X and Y directions evaluated on the low-noise and high-noise test sets, respectively; where (a) is the absolute error of the feature point regression in the X direction of the DETR model evaluated on the low-noise test set; (b) is the absolute error of the feature point regression in the Y direction of the DETR model evaluated on the low-noise test set; (c) is the absolute error of the feature point regression in the X direction of the DETR model tested on the high-noise test set; (d) is the absolute error of the feature point regression in the Y direction of the DETR model tested on the high-noise test set;

[0040] Fig.10 Part of the prediction results obtained by three different models on the test set. DETAILED DESCRIPTION

[0041] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0042] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0043] Embodiment 1

[0044] In one or more embodiments, a noise-resistant weld feature recognition method based on a lightweight neural network is disclosed, which specifically includes the following process:

[0045] (1) Obtain welding image data during the welding process;

[0046] (2) Input the acquired welding image data into the trained weld feature recognition model and output the type and location information of the weld;

[0047] Among them, the weld feature recognition model includes a backbone network, a neck layer and a head layer; the backbone network outputs feature information of different scales and transmits the feature information to the neck layer; the neck layer fuses the feature information of different scales, and the fused feature map is input into the head layer; the head layer includes three branches: a heat map branch, an offset branch and a classification branch. The heat map branch is used to output a heat map tensor to obtain the probability that each eigenvalue represents the probability that the corresponding area of ​​the point contains a feature point; the offset branch is used to output an offset tensor to obtain the position deviation of the feature points generated in each dimension; the classification branch is used to output a classification tensor to obtain the corresponding score of each weld type; and finally the weld position and weld type (such as Y-type weld, Lap lap weld, Butt butt weld, etc.) are predicted.

[0048] Specifically, the backbone network extracts feature information from the image through a multi-layer neural network stack, and passes feature information of different scales and extraction degrees into the neck layer. In order to improve the processing speed of the network and enable the network to run better on non-GPU devices, this embodiment uses a lightweight backbone architecture MobileNetv3 as the backbone network part, which effectively reduces network parameters. MobileNetV3 can output activation features of different levels, and input activation features of different levels into the neck layer for further image processing tasks and subsequent image processing.

[0049] At the neck layer, the feature pyramid network (FPN) structure is applied to fuse feature information of different scales. Specifically, each stage of the backbone network will generate feature maps of different resolutions and semantic levels. For example, the shallow network is mainly based on the morphological features of the image and is more sensitive to position information; the deep network is mainly based on the semantic features of the image and is more sensitive to information such as the category of welds and noise recognition. Therefore, this embodiment extracts three branches in the backbone network. As shown in Figure 1, FPN will fuse the backbone network layer with a downsampling step size of {8, 16, 32} relative to the input image, and integrate the deep abstract semantic information and the surface coarse-grained feature information to obtain better small-scale feature detection capabilities.

[0050] Combination Figure 1 The specific structure of the neck layer in this embodiment includes:

[0051] The deepest-scale feature information output by the backbone network passes through the cascaded channel attention module, the third convolutional layer, and the adaptive pooling layer in sequence, and then is output to the classification branch of the head layer;

[0052] The feature information of the second scale output by the backbone network passes through the second convolutional layer, is merged with the output of the upsampled third convolutional layer, and then output to the first DP module;

[0053] The feature information of the first scale output by the backbone network passes through the first convolutional layer, is merged with the output of the upsampled first DP module, and is transmitted to the heat map branch and offset branch of the head layer respectively through the second DP module.

[0054] In this embodiment, a DP block is designed to replace some standard 3 × 3 convolution blocks in the neck layer to obtain better lightweight performance. Figure 2 , the structure of the DP block includes a deep convolutional layer (DWC), the first BN layer (normalization layer), an activation function layer, a point-by-point convolutional layer (PWC), and a second BN layer connected in sequence; spatial feature information is obtained through the deep convolutional layer, and on top of the spatial feature information extracted by the deep convolutional layer, these spatial features are combined through the point-by-point convolutional layer to achieve the fusion of spatial feature information. After each convolution process, the BN layer (normalization layer) follows. The BN layer normalizes each feature channel of each small batch of data so that its mean is close to 0 and its variance is close to 1, thereby reducing the internal covariate shift. In addition, after the DWC+BN layer processing, the ReLU6 activation function that is friendly to low-dimensional feature information is added. The ReLU6 activation function can introduce nonlinearity to the neural network, enabling the network to learn and perform more complex tasks, allowing the network to simulate more complex function mappings, and increasing the expressiveness and flexibility of the model; at the same time, it can alleviate the gradient vanishing problem, making the training process more stable and accelerating the convergence speed.

[0055] In this embodiment, spatial feature information is obtained through a deep convolution layer, where the spatial feature information mainly refers to the geometric shape and structure of the weld in the image, including the width, length, shape (straight line, curve, branch), etc. of the weld, as well as some feature information such as the boundary between the weld and the surrounding material. Channel information, that is, the association between different feature maps, can be obtained through a point-by-point convolution layer.

[0056] By introducing the DP block, the standard convolution operation is decomposed into depthwise convolution and point-by-point convolution, which significantly reduces the number of model parameters and computational complexity. As a result, the memory occupied by the model at runtime is also reduced accordingly, reducing the performance requirements for mobile and embedded deployment devices.

[0057] In addition, this embodiment also makes some improvements to the architecture of the neck layer. Compared with the original method shown in Figure 3 (a), the improved top-down pathway and lateral connection in this embodiment adopt a serial rather than additive method, as shown in Figure 3 (b), and the features of each input branch are directly merged through the Concat function. Adjusting the channel helps to merge the feature map output by PWC with the upsampling result of the coarser resolution feature map containing more semantic information. Finally, the neck layer will generate a feature map of size 128 × 128 × 64, which is then input to the head layer.

[0058] In addition, in the neck layer, the network connects a self-designed cascade channel attention module (CCAM) with 32×n (downsampled output), aiming to enhance the model's noise resistance.

[0059] The deepest output shape of the backbone network obtained at the neck layer is Hin × Win × Cin, specifically 16 × 16 × 512. It should be noted that a considerable number of features are related to noise. These features will be merged at the neck layer and ultimately affect the output of the head layer. In order to reduce the impact of noise, CCAM is used to filter the noise. Figure 4 As shown in the figure, CCAM divides the deepest scale feature information output by the backbone network into three blocks, each of which is channel-weighted by the channel attention module; the outputs of the three blocks after channel weighting are spliced ​​to obtain the output of the cascade channel attention module; among them, the output of the first block after the channel attention module is merged with the second block; the output of the second block after the channel attention module is merged with the third block.

[0060] CCAM uses the channel attention (CA) module as the core feature selector, uses the global average pooling layer to squeeze the input feature map into a vector, and then sends it to the first fully connected (FC) layer for feature dimension reduction, and then sends it to a subsequent FC layer to restore the feature number. The resulting vector is mapped to the range of 0 to 1 through the softmax function, and the input feature map channel is weighted through the channel multiplication operator. The higher the weight of any channel, the more important the features in the channel. This embodiment uses the softmax function for the vector generated by FC instead of using the sigmoid function for the entire feature map; this replacement can reduce the amount of calculation with almost no loss of accuracy. The calculation formula for the refined feature map FCA of the CA module is as follows:

[0061] (1)

[0062] Among them, z represents the input feature, and represents the parameters of the two FC stages, r represents the reduction rate of 16 in the neural network, C represents the number of input tensors of CCAM, and g represents the number of groups of the input tensors.

[0063] In CCAM, the output of the first CA block is combined with the output of the second block by element-by-element addition to enhance the information fusion between channels. For CCAM, the computational effort generated by the FC layer is The FC layer does not rely on the cascade grouping structure, but directly uses channel attention for calculation, and the amount of calculation is When the cascade grouping structure is used, the computational effort of the CA module is approximately .

[0064] At the head layer, the position of the feature points and the weld type are predicted. For each weld image, the weld type is single and the key point type is the same. This embodiment designs the following Figure 1 The heat map branch, offset branch and separate classification branch are shown. The heat map branch is used to output the heat map tensor, which is a 128*128*1 feature vector map. Each eigenvalue represents the probability that the area corresponding to the point contains the feature point. The offset branch is used to output the offset tensor. The offset tensor is a 128*128*2 displacement map. Each dimension generates the deviation of the detailed position of the feature point relative to the upper left corner of the area, which is used to eliminate the quantization error of the single heat map prediction. The classification branch is used to output the classification tensor. The classification branch has been redesigned and is derived from the deepest network layer of the backbone network. Through convolution, pooling and other operations, it finally generates a 1*1*3 classification feature vector, obtains the corresponding scores of the three weld categories, and judges the weld type according to the scores.

[0065] The heat map tensor of this embodiment is usually used to indicate the position of the weld in the image, and the offset tensor can provide a finer position adjustment. Combining these two types of information can more accurately locate the weld. The classification tensor can provide information about the type of weld. Combined with the heat map and offset tensor, the type of weld can be identified while locating the weld. The use of the heat map tensor and the offset tensor improves the interpretability of the model prediction and helps to understand how the model makes predictions.

[0066] By considering multiple partitions (i.e., different regions in the heatmap tensor) and selecting the top two with the highest scores, the model can better handle weld discontinuities or noise in the image, reducing the likelihood of misidentifying non-weld areas as welds. If the highest-scoring partition in the heatmap is inaccurate due to some noise, the next highest-scoring partition can be considered as an alternative, thereby increasing the overall system's tolerance to incorrect predictions. Because the bias tensor can fine-tune the predictions in the heatmap, it allows the model to adapt to welds of different shapes and sizes.

[0067] The specific loss function of the weld feature recognition model in this embodiment is as follows:

[0068] The head layer produces three parts, namely the heat map tensor ∈ , bias tensor and categorical tensors (i.e. one-hot ncoding) , where W and H are the input image sizes of the model, c is the total number of weld types, and R is the feature map scaling factor corresponding to each head layer.

[0069] Throughout the training process, the loss is determined by comparing the predicted values ​​generated by the head layer with the ground truth label values, emphasizing that the shape of the label value should match the shape of the prediction. This comparison aims to optimize the neural network parameters so that the model outputs the best score or regression value prediction. Therefore, in this case, the generation of labels must be solved.

[0070] Label Heatmap is a method to represent the location of objects in an image. It maps label (category) information to each pixel of the image in the form of color coding to form a heatmap of the same size as the original image. In this heatmap, different colors represent different labels or categories, so that the category to which each pixel belongs can be intuitively displayed.

[0071] The output heat map tensor in this embodiment is the generated label heat map, where each value represents the probability that the corresponding pixel belongs to a certain category. The coordinates of the feature point position are mapped to the corresponding partition in the heat map tensor label as follows: , is the label value of the kth feature point coordinate, which is related to the size of the model input image (that is, the input size of this algorithm is 512×512), R = 4.

[0072] On the label heat map, positive sample points are represented by Gaussian circles: To accelerate the convergence of the model, is a factor that varies with the size of the object, Represents the center of the Gaussian circle, which is the ideal positive position where the feature point is mapped on the heat map.

[0073] Given the inherent rounding effect of each layer division, relying solely on heatmaps to recover the location information of feature points may result in reduced accuracy. To solve this problem, an offset is introduced. The offset label value of each feature point is expressed as:

[0074] (2)

[0075] in, is a two-dimensional coordinate representing the relative position ratio of the feature point in the corresponding partition of the label heatmap. Therefore, the offset is represented by a two-channel tensor. In addition, the offset tensor bins are still uniquely valid in the corresponding bins of the positive positions in the heatmap tensor.

[0076] set up Location in the heatmap produced for the model The prediction score at position (i, j) represents the probability or confidence that the model predicts that the pixel at that position belongs to each category. The parameters for generating the heat map are optimized using a variant of the focal loss, which is expressed as:

[0077] (3)

[0078] Among them, M ij For location The true label at the position indicates the probability that the center point of the object appears at this position, M ij The value of is used to train the model and can help the model learn to predict the location of objects. N0 is the number of feature points in the structured light streak image, H and W represent the spatial size of the heat map, α is used to fine-tune the weights of challenging and easy-to-locate points, and β controls the weights of non-central values ​​within the Gaussian circle. It is worth noting that as the distance between the label and the center of the Gaussian circle increases, the value assigned to The loss weight of the corresponding partition prediction score will also increase.

[0079] The essence of offset prediction originates from the regression task. In order to achieve accurate offset prediction, the following smooth L1 loss function is used:

[0080] (4)

[0081] Where N0 is the number of feature points in the structured light fringe image, Indicates the offset label value of each feature point, Represents the offset prediction value. Smooth L1 loss takes advantage of the advantages of L1 and L2 loss functions. This method can prevent gradient explosion in the initial stage of training and promote gentler gradients during the backpropagation process after training. This greatly helps improve the convergence of the model.

[0082] Cross entropy loss can be used for weld classification tasks. Here, let Represents the score of the i-th type of structured light stripe image predicted by the model of this embodiment, a Boolean variable Represents the label value of the i-th category in single-point encoding , the loss caused by classification is expressed by the following formula:

[0083] (5)

[0084] where c represents the total number of weld types. Finally, the total loss is the overall training objective, and its derivation process is as follows:

[0085] (6)

[0086] in, , and They are heat map loss, offset loss and classification loss respectively. and Represents the constant weight assigned to bias loss and classification loss, and the configuration value is 1.

[0087] There is a correlation between the point count prediction and the laser stripe classification. Specifically, if the classification result is predicted as "lap seam", the model identifies the top two partitions in the heatmap tensor with the highest prediction scores. This data is then combined with the corresponding segments on the offset tensor to determine the final localization prediction result. In this case, the suppression effect of the prediction score threshold is not considered. However, the heatmap tensor is resized to match the uniform shape of the original image and completely covers the original image. The colors in the heatmap represent the prediction scores assigned to potential feature points. Figure 5 A feature point recognition effect diagram is given, which shows a broken laser stripe. Although the two ends are very close, the implementation method can still accurately identify the two ends as potential feature points.

[0088] The method of this embodiment is verified as follows:

[0089] The dataset used to train and test the model comes from the weld location and tracking sensor mentioned above. The model is trained on a computer equipped with an Nvidia 2080Ti GPU and an Intel Xeon(R) E5-2683 CPU.

[0090] The backbone layers are initialized with the weights of the pre-trained MobileNetv3 to speed up training and achieve fast convergence. The FC layer is implemented using a standard 1 × 1 convolution, and the convolution layer parameters are initialized using the He initialization method. As for the BN layer parameters, the weights are initialized to 1 and the biases are initialized to 0. Adam is used as the optimizer, and the initial learning rate is set to 5×10 -4 .

[0091] For welding tasks, the weld feature recognition model of this embodiment model (Ours) can only predict one category of an image related to all points. DETR can learn the correlation between the welding image category and the feature point classification in the dataset, thereby preventing the points in a single image from being mistakenly classified into different categories. Each point inferred by YOLOv5n will be matched to one category.

[0092] In order to compare the performance of several models, the category of the point with the highest predicted classification confidence in each image is the category of the entire image. In order to simplify the comparison and solve the difference in sample quantity, the following three indicators are used to evaluate the three models in weld classification:

[0093] (7)

[0094] (8)

[0095] (9)

[0096] in, Represent the weighted average precision, recall and F1-score respectively. are the precision, recall and F1-score of the i-th category respectively. Represents the weight coefficient of the i-th category (1 / N in this embodiment).

[0097] True positive (TP) indicates the number of samples predicted correctly, false negative (FN) indicates the number of labeled boxes without corresponding correct prediction boxes, and false positive (FP) indicates the number of prediction boxes that do not meet the correct prediction criteria, which can be calculated from the confusion matrix. The subscript i indicates the i-th category, for example: It indicates the number of samples with correct prediction for the i-th category.

[0098] The index values ​​of the three models on the adjusted test set are shown in Table 1. Figure 6 (a)-Figure 6 (c) show the confusion matrices obtained by different models on the adjusted test set. Figure 6 (a)-Figure 6 (c) correspond to the prediction results of the model of this embodiment, YOLOv5n and DETR respectively.

[0099] It is worth noting that among the three key indicators for evaluating classification performance, the model of this embodiment has the highest value, with a weighted average precision of 0.9674, a weighted average recall of 0.9666, and a weighted average F1 score of 0.9668.

[0100] Table 1 Index values ​​of three models

[0101]

[0102] In order to evaluate the feature point positioning performance of the model in this embodiment under different noise levels and evaluate its anti-noise ability. The performance indicators include mean absolute error (MAE), root mean square error (RMSE), standard deviation , and the projection of the predicted and label coordinates to find the average Euclidean distance relative to the camera coordinate system .

[0103] The positioning error of the feature point is calculated according to the X-axis and Y-axis directions of the image as follows:

[0104] (10)

[0105] in, and Represent the 2D label coordinates and 2D predicted coordinates of the feature points respectively.

[0106] MAE is used to evaluate the static accuracy of positioning, and its derivation process is as follows:

[0107] (11)

[0108] Where N represents the total number of feature points in the specified test set, which is consistent with the definition in the following formula. The root mean square error will highlight the impact of predicted outliers, and its derivation process is as follows:

[0109] (12)

[0110] Used to evaluate the stability / robustness of the model, the derivation process is as follows:

[0111] (13)

[0112] in, All feature points The average value of .

[0113] After projecting the two-dimensional positioning results into three-dimensional space, the measurement indicators are used To evaluate the error level of the three models. The calculation formula is as follows:

[0114] (14)

[0115] in, and Represent the 3D label coordinates and 3D predicted coordinates of the feature points respectively. According to the formula and The coordinates of the feature points relative to the camera coordinate system can be calculated. This embodiment a priori assumes that the calibration results of the line structured light plane equation and the camera intrinsic matrix are ideal, and any error can only be attributed to the position of the feature points predicted by the feature point positioning model in the image coordinate system.

[0116] Figure 7 , Figure 8 and Fig. 9 The absolute error curves of the model in this embodiment, YOLOv5, and DETR (Detection Transformer, an object detection model based on the Transformer architecture) are shown respectively, which represent the test results using the three models. Among them, the green curve corresponds to the absolute error measured by the low-noise test set, the blue curve corresponds to the absolute error measured by the high-noise test set, and the red horizontal line represents the mean absolute error (MAE).

[0117] in, Figure 7 (a) and (b) in FIG. 5 respectively illustrate the absolute value errors of feature point regression in the X and Y directions of the model of this embodiment evaluated on a low-noise test set. Figure 7 (c) and (d) in FIG. 5 show the absolute error of feature point regression in the X and Y directions of the model of this embodiment tested on a high noise test set. The peak value, the peak position of the absolute error, and the mean absolute error (MAE) are all marked. Figure 8 (a) and (b) in Figure 5 illustrate the absolute error of feature point regression in the X and Y directions for the YOLOv5 model evaluated on the low-noise test set, respectively. Figure 8 (c) and (d) in Figure 1 show the absolute error of feature point regression in the X and Y directions of the YOLOv5 model tested on the high noise test set. The peak value, the peak position of the absolute error, and the mean absolute error (MAE) are all marked. Fig. 9 (a) and (b) in Figure 3 illustrate the absolute value error of feature point regression in the X and Y directions for the DETR model evaluated on the low-noise test set, respectively. Fig. 9 (c) and (d) in Figure 3 show the absolute error of feature point regression in the X and Y directions of the DETR model tested on the high noise test set. The peak value, the peak position of the absolute error, and the mean absolute error (MAE) are all marked.

[0118] All indicators related to feature point positioning error have been calculated and listed in Tables 2 and 3, including mean absolute error (MAE), root mean square error (RMSE) and standard deviation. The image is evaluated in the X and Y directions and the average values ​​in the two directions are calculated. At the same time, Table 2 and Table 3 also list the results of the model of this embodiment without using CCAM.

[0119] Table 2 Statistical characteristics of the three models for positioning the feature points of low-noise welding structured light stripes

[0120]

[0121] In Table 2, Ours represents the model of this embodiment, and Ours† represents the model of this embodiment without using CCAM. MAE, RMSE, and σf are all in pixels. The unit is millimeters.

[0122] It can be concluded that, in the case of low noise, the MAE and RMSE of the three models are basically at the same level. It is worth noting that in terms of average σf, the models of this embodiment with and without CCAM perform outstandingly compared with the other two models, reaching 1.943 pixels and 1.922 pixels respectively. In terms of the performance, the model of this embodiment performs best, reaching 0.197 mm. Under low noise test conditions, using CCAM does not produce significant differences.

[0123] Table 3 Experimental results of evaluating the positioning performance of weld feature points in the high noise test set

[0124]

[0125] In Table 3, Ours represents the model of this embodiment, and Ours† represents the model of this embodiment without using CCAM. MAE, RMSE and σf are all in pixels. The unit is millimeters.

[0126] It can be concluded that in a noisy environment, the average MAE, RMSE and ρ of the model in this embodiment are meanare 1.736 pixels, 2.407 pixels, and 0.205 mm, respectively. These indicators are better than YOLOv5 and DETR. In addition, compared with the model of this embodiment without CCAM, the model integrated with CCAM shows higher performance in most positioning indicators. This phenomenon is in sharp contrast to the measurement results observed in the low-noise test set. The average σ of the model of this embodiment f The value is 2.217 pixels, which shows that the model of this embodiment has excellent stability in accurately locating welding feature points in high-noise welding images.

[0127] The lightweight effect of the model of this embodiment is verified below.

[0128] The lightweight characteristics of the model in this embodiment are evaluated based on multiple indicators, including the total number of model parameters (Params), the number of floating-point operations (FLOPs), the average latency (Mean latency), and the frames per second (FPS). Params can indirectly measure the computational complexity and memory utilization, while FLOPs represents the computational cost of the model. Lower average latency (or higher frames per second) means reduced inference time, which is a key factor in achieving optimal real-time performance of the weld tracking system using this model. This is especially important for complex weld tracking applications, welding speed, and welding process control. The average latency of the three models was determined by performing 100 tests using a single image, and the frames per second (FPS) value was calculated by dividing the average latency by 1000 milliseconds. Table 4 provides an overview of the lightweight indicators of the three models: the model in this embodiment, YOLOv5n, and DETR. Fig.10 The prediction results obtained by different models on the test set are given.

[0129] Table 4 Comparison of lightweight indicators of three models on Intel Core i7-8650U CPU

[0130]

[0131] It can be seen that the model of this embodiment outperforms other models in all indicators. Params is 1.3 M, equivalent to 72% of YOLOv5n and 3.5% of DETR; GFLOPs is 0.87 GFLOPs, equivalent to 21% of YOLOv5n and 1.5% of DETR. With mean latency of 29.32 milliseconds and FPS of 34.11, the model of this embodiment can meet the strict requirements of real-time weld tracking.

[0132] Embodiment 2

[0133] In one or more embodiments, a noise-resistant weld feature recognition system based on a lightweight neural network is disclosed, comprising:

[0134] A data acquisition module, used for acquiring welding image data during the welding process;

[0135] The weld prediction module is used to input the acquired welding image data into the trained weld feature recognition model and output the type and location information of the weld;

[0136] Among them, the weld feature recognition model includes a backbone network, a neck layer and a head layer; the backbone network outputs feature information of different scales and transmits the feature information to the neck layer; the neck layer fuses the feature information of different scales, and the fused feature map is input into the head layer; the head layer includes three branches: a heat map branch, an offset branch and a classification branch. The heat map branch is used to output a heat map tensor to obtain the probability that each eigenvalue represents the corresponding area of ​​the point containing the feature point; the offset branch is used to output an offset tensor to obtain the position deviation of the feature point generated in each dimension; the classification branch is used to output a classification tensor to obtain the corresponding score of each weld type; and finally the position and type of the weld are predicted.

[0137] The specific implementation of the above modules is the same as that in Example 1 and will not be described in detail.

[0138] Embodiment 3

[0139] In one or more embodiments, a terminal device is disclosed, including a server, wherein the server includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the anti-noise weld feature recognition method based on a lightweight neural network in Example 1 is implemented.

[0140] Embodiment 4

[0141] In one or more embodiments, a computer-readable storage medium is disclosed, in which a plurality of instructions are stored, wherein the instructions are suitable for being loaded by a processor of a terminal device and executing the noise-resistant weld feature recognition method based on a lightweight neural network described in Example 1.

[0142] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A noise-resistant weld feature recognition method based on lightweight neural network, characterized in that: include: Acquire welding image data during welding process; Input the acquired welding image data into the trained weld feature recognition model, and output the type and location information of the weld; Among them, the weld feature recognition model includes a backbone network, a neck layer and a head layer; the backbone network outputs feature information of different scales and transmits the feature information to the neck layer; the neck layer fuses the feature information of different scales, and the fused feature map is input into the head layer; the head layer includes three branches: a heat map branch, an offset branch and a classification branch. The heat map branch is used to output a heat map tensor to obtain the probability that each area contains a feature point; the offset branch is used to output an offset tensor to obtain the position deviation of the feature point generated in each dimension; the classification branch is used to output a classification tensor to obtain the corresponding score of each weld type; and finally the weld position and weld type are predicted; The neck layer adopts a feature pyramid network structure, including: the deepest scale feature information output by the backbone network passes through the cascade channel attention module, the third convolution layer and the adaptive pooling layer in sequence, and is output to the classification branch of the head layer; the second scale feature information output by the backbone network passes through the second convolution layer, is merged with the output of the upsampled third convolution layer, and then is output to the first DP module; the first scale feature information output by the backbone network passes through the first convolution layer, is merged with the output of the upsampled first DP module, and is transmitted to the heat map branch and the offset branch of the head layer respectively through the second DP module; The cascade channel attention module divides the deepest scale feature information output by the backbone network into three blocks, and each block is channel-weighted by the channel attention module; the outputs of the three blocks after channel weighting are spliced ​​to obtain the output of the cascade channel attention module; wherein the output of the first block after passing through the channel attention module is merged with the second block; the output of the second block after passing through the channel attention module is merged with the third block; The first DP module and the second DP module have the same structure, including: a deep convolution layer, a first BN layer, an activation function layer, a point-by-point convolution layer and a second BN layer connected in sequence; spatial feature information is obtained through the deep convolution layer, and on the spatial feature information extracted by the deep convolution layer, these spatial features are combined through the point-by-point convolution layer to achieve fusion of spatial feature information.

2. The noise-resistant weld feature recognition method based on lightweight neural network according to claim 1 is characterized in that: The backbone network adopts a lightweight MobileNetv3 structure.

3. The noise-resistant weld feature recognition method based on lightweight neural network according to claim 1 is characterized in that: The loss function of the weld feature recognition model is: ； in, , and They are heat map loss, offset loss and classification loss respectively. and are the weights assigned to bias loss and classification loss, respectively.

4. The noise-resistant weld feature recognition method based on lightweight neural network as claimed in claim 3 is characterized in that: The heat map loss is specifically: ； Among them, assuming Locations in the heat map output for the weld feature recognition model The prediction score at M ij For location The true label at , which indicates the probability that the center point of the object appears at this position; N0 is the number of feature points in the image, H and W represent the spatial size of the heat map, and α and β are weights.

5. A noise-resistant weld feature recognition system based on lightweight neural network, characterized in that: include: A data acquisition module, used for acquiring welding image data during the welding process; The weld prediction module is used to input the acquired welding image data into the trained weld feature recognition model and output the type and location information of the weld; Among them, the weld feature recognition model includes a backbone network, a neck layer and a head layer; the backbone network outputs feature information of different scales and transmits the feature information to the neck layer; the neck layer fuses the feature information of different scales, and the fused feature map is input into the head layer; the head layer includes three branches: a heat map branch, an offset branch and a classification branch. The heat map branch is used to output a heat map tensor to obtain the probability that each area contains a feature point; the offset branch is used to output an offset tensor to obtain the position deviation of the feature point generated in each dimension; the classification branch is used to output a classification tensor to obtain the corresponding score of each weld type; and finally the weld position and weld type are predicted; The neck layer adopts a feature pyramid network structure, including: the deepest scale feature information output by the backbone network passes through the cascade channel attention module, the third convolution layer and the adaptive pooling layer in sequence, and is output to the classification branch of the head layer; the second scale feature information output by the backbone network passes through the second convolution layer, is merged with the output of the upsampled third convolution layer, and then is output to the first DP module; the first scale feature information output by the backbone network passes through the first convolution layer, is merged with the output of the upsampled first DP module, and is transmitted to the heat map branch and the offset branch of the head layer respectively through the second DP module; The cascade channel attention module divides the deepest scale feature information output by the backbone network into three blocks, and each block is channel-weighted by the channel attention module; the outputs of the three blocks after channel weighting are spliced ​​to obtain the output of the cascade channel attention module; wherein the output of the first block after passing through the channel attention module is merged with the second block; the output of the second block after passing through the channel attention module is merged with the third block; The first DP module and the second DP module have the same structure, including: a deep convolution layer, a first BN layer, an activation function layer, a point-by-point convolution layer and a second BN layer connected in sequence; spatial feature information is obtained through the deep convolution layer, and on the spatial feature information extracted by the deep convolution layer, these spatial features are combined through the point-by-point convolution layer to achieve fusion of spatial feature information.

6. A terminal device, comprising a processor and a memory, wherein the processor is used to implement instructions; and the memory is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the noise-resistant weld feature recognition method based on a lightweight neural network as described in any one of claims 1-4.

7. A computer-readable storage medium storing a plurality of instructions, characterized in that: The instructions are suitable for being loaded by a processor of a terminal device and executing the noise-resistant weld feature recognition method based on a lightweight neural network as described in any one of claims 1-4.

Images