An automatic detection method for steel welds based on cascaded convolutional neural networks

Abstract

The present invention provides a steel weld automatic detection method based on a cascade convolutional neural network, comprising the following steps: S1, scanning the steel using an industrial camera and an infrared laser line to obtain weld data; S2, marking the weld area and center line of the image to complete the preparation of a weld reference image set and a center line reference image set; S3, training the cascade convolutional neural network using the weld data and the weld reference image set according to the training algorithm, and training the center line extraction network using the weld data and the center line reference image set; S4, scanning the steel to be detected using an industrial camera and an infrared laser line to obtain weld data to be detected; S5, inputting the weld data to be detected into a trained detection model to output the detection result. The present invention can accurately extract the weld position, greatly improve the anti-interference ability, ensure the welding quality, and improve the adaptive ability of the automatic welding system.

Description

Technical Field

[0001] The present invention belongs to the technical field of weld detection, and in particular relates to an automatic steel weld detection method based on a cascaded convolutional neural network. Background Art

[0002] Welding technology has become one of the most widely used connection methods, mainly used in the fields of aerospace, electronics manufacturing, machinery manufacturing, and shipbuilding. However, the welding environment is very harsh. The harmful gases and dazzling arc light produced by the welding torch during welding can easily threaten the lives of welders. With the aging of the population, the number of welders is decreasing year by year, while the demand for welding is increasing year by year. Traditional manual welding cannot meet the needs of the current society.

[0003] With the development of artificial intelligence industry and electronic manufacturing technology, new ideas and new technologies have been provided for welding industry. Artificial intelligence will provide stronger technical support for the realization of welding automation and intelligence, and can meet the welding needs of the new era. Weld detection technology is the key technology to realize welding automation and intelligence. Efficient and accurate weld identification and detection has great engineering practical significance for the realization of the third generation of autonomous intelligent welding robots.

[0004] With the rapid development of automation in the manufacturing industry, metal welding has become an indispensable part of the production process. In order to meet the requirements of automated welding production, it is necessary to obtain real-time position information such as the width and center line of the weld during the welding process. Weld tracking based on laser vision has become a research hotspot in the field of automatic welding. Laser vision sensing technology projects laser onto the weld surface to form a weld stripe image containing weld contour information, and then performs feature analysis. During the welding process, due to interference from strong arc light, spatter, and arc noise, the collected image is seriously polluted by noise and is difficult to accurately locate, which directly affects the quality of welding. Therefore, in the process of automatic welding, it is particularly important to accurately obtain weld position information.

[0005] There are many methods for extracting weld laser line features, including the threshold method and grayscale centroid method for segmenting the target and background at different grayscale values, the edge method for extracting the edge of the target and the geometric center method for extracting the center of the image, and the use of the Steger algorithm to find the pixel position of the light stripe. The weld detection method designed for a specific type of steel can achieve high-precision detection. However, the detection accuracy of welds of different types of steel parts with strong arc interference is low, which is difficult to meet the needs of actual welding tracking. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an automatic detection method for steel welds based on a cascaded convolutional neural network to solve the technical problem that the weld detection method designed for a specific type of steel can achieve high-precision detection, while improving the low detection accuracy of steel welds of different categories with strong arc interference. The present invention has high accuracy when detecting welds of different categories of steel parts, and can accurately extract the weld position, center line position and feature point position under interference from strong arc light, spatter and arc noise, and has the advantages of strong anti-interference ability and accurate recognition.

[0007] To achieve the above object, the present invention provides the following technical solution: a method for automatic detection of steel welds based on a cascaded convolutional neural network, comprising the following steps:

[0008] Step S1, obtaining weld data, scanning the steel part using an industrial camera and an infrared laser line to obtain weld data;

[0009] Step S2, data annotation, annotating the weld area and center line of the image, and completing the preparation of the weld reference image set and the center line reference image set;

[0010] Step S3, training the cascade convolutional neural network and the centerline extraction network, according to the training algorithm, using the weld data and the weld reference image set to train the cascade convolutional neural network, and using the weld data and the centerline reference image set to train the centerline extraction network;

[0011] Step S4, obtaining the data of the weld to be detected, scanning the steel part to be detected using an industrial camera and an infrared laser line to obtain the data of the weld to be detected;

[0012] Step S5, calculating the detection result, inputting the weld data to be detected into the trained detection model to output the detection result.

[0013] Furthermore, the step S2 specifically includes:

[0014] Step S2-1, converting the weld data image into PNG format;

[0015] Step S2-2, annotating the weld image processed by step S2-1, wherein the annotated content includes the position and size of the weld area, and the position and size of the center line.

[0016] Furthermore, in step S3, the training algorithm includes a stochastic gradient training algorithm, an Adam algorithm, an RMSProp algorithm, an Adagrad algorithm, an Adadelta algorithm or an Adamax algorithm.

[0017] Furthermore, in step S3, the cascaded convolutional neural network includes an input layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a fifth deconvolutional layer, a sixth convolutional layer, a fourth deconvolutional layer, a seventh convolutional layer, a third deconvolutional layer, an eighth convolutional layer, a second deconvolutional layer, a first deconvolutional layer, a ninth convolutional layer, an output layer and a Softmax layer in sequence; low-layer convolution can extract more specific details, including laser stripe edge information, while high-layer convolution can extract more position information, and the low-layer and high-layer outputs are added to perform feature fusion, and local detail features are supplemented by feature fusion; in the cascaded convolutional neural network, the outputs of the first convolutional layer and the second deconvolutional layer, the second convolutional layer and the third deconvolutional layer, the third convolutional layer and the fourth deconvolutional layer, and the fourth convolutional layer and the fifth deconvolutional layer are feature fused.

[0018] Furthermore, in step S3, in order to train the cascade convolutional neural network, a cross entropy loss is used, and the calculation method is shown in Formula 1;

[0019]

[0020] Where θ1 represents the parameters of the cascaded convolutional neural network, M represents the minimum batch size, N represents the number of pixels in the block, and K represents the number of categories; 1(y=k) is a function that takes 1 when y=k and 0 otherwise; represents the jth pixel in the i-th block, Indicates that the last convolutional layer is Pixel output; express The pixel to which the pixel corresponds in the true label;

[0021] The probability that a pixel belongs to the kth class is The calculation method is shown in formula 2;

[0022]

[0023] Furthermore, in step S3, the centerline extraction network includes an input layer, a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a third convolutional layer, a third pooling layer, a third upper pooling layer, a third deconvolutional layer, a second upper pooling layer, a second deconvolutional layer, a first upper pooling layer, a first deconvolutional layer, an output layer, and a Softmaxc layer in sequence; compared with the cascaded convolutional neural network, the centerline extraction network also includes an encoding layer and a decoding layer, but the centerline extraction network structure is smaller; on the one hand, the feature map output by the last convolutional layer of the cascaded convolutional neural network contains less interference information than the original image, and the centerline extraction network can use this feature map as the input of the network; on the other hand, compared with weld detection, only fewer centerline pixels are needed to train the centerline extraction network.

[0024] Furthermore, in step S3, in order to train the centerline extraction network, a cross entropy loss is used, and the calculation method is shown in Formula 3;

[0025]

[0026] Where θ2 represents the parameters of the centerline extraction network, 1(z=k) is a function that takes 1 when z=k and 0 otherwise; Represents the output of the last deconvolution layer of the center extraction network; express The pixel corresponds to the pixel in the centerline ground truth label;

[0027] The probability that a pixel belongs to the kth class is The calculation method is shown in formula 4;

[0028]

[0029] Furthermore, in step S3, the overall network loss is calculated as shown in Formula 5;

[0030] Loss(θ1, θ2) = loss seg (y,f(x),θ1)+loss cen (z,h(x),θ2) Equation 5.

[0031] Furthermore, the step S5 specifically includes:

[0032] Step S5-1, using the grayscale centroid method to refine the center line;

[0033] Step S5-2, for single-pass welding, the least square method LSM is used to fit the center line, and the intersection of the center line is the position of the feature point; for multi-pass welding, NURBS is used to fit the center line curve, and the position of the feature point is obtained by derivation of the fitting curve.

[0034] Furthermore, in step S5-2, the NURBS curve is calculated as shown in Formula 6;

[0035]

[0036] Where P i represents the control vertex, w i represents the weight factor, a and b are 0 and 1 respectively, N i，p (u) is the P-order B-spline basis function defined on the knot vector U;

[0037] The calculation method of the node vector U is shown in Equation 7;

[0038]

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] (1) It can accurately extract the weld position under interference from strong arc light, spatter, and arc noise. This method greatly improves the anti-interference ability, ensures the welding quality, and improves the adaptability of the automatic welding system.

[0041] (2) By performing feature fusion to supplement local detail features, no pre-processing or post-processing is required, making the laser stripe boundary more accurate and clearer.

[0042] (3) The cascaded convolutional neural network has deep learning capabilities. It continuously trains and learns the characteristic information of the laser line, integrates the multi-level features of the weld image, and can accurately obtain the characteristics of the entire weld. At the same time, it also has good noise suppression capabilities in terms of details, and has better extraction effect and extraction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is the overall flow chart of the present invention;

[0044] Figure 2 A schematic diagram of the network structure of the cascaded convolutional neural network and the centerline extraction network of the present invention;

[0045] Figure 3 It is a schematic diagram of the position of the characteristic points of a single pass welding;

[0046] Figure 4 Schematic diagram of the locations of characteristic points of multi-pass welding. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. The specific embodiments described here are only used to explain the technical solution of the present invention and are not limited to the present invention.

[0048] An automatic detection method for steel welds based on cascaded convolutional neural networks, such as Figure 1 As shown, the following steps are included:

[0049] Step S1, obtaining weld data, scanning the steel part using an industrial camera and an infrared laser line to obtain weld data;

[0050] Step S2, data annotation, annotating the weld area and center line of the image, and completing the preparation of the weld reference image set and the center line reference image set;

[0051] Step S2 specifically includes:

[0052] Step S2-1, converting the weld digital image into PNG format;

[0053] Step S2-2, annotating the weld image processed by step S2-1, wherein the annotated content includes the position and size of the weld area, and the position and size of the center line.

[0054] Step S3, training the cascade convolutional neural network and the centerline extraction network, according to the training algorithm, using the weld data and the weld reference image set to train the cascade convolutional neural network, and using the weld data and the centerline reference image set to train the centerline extraction network;

[0055] Step S4, obtaining the data of the weld to be detected, scanning the steel part to be detected using an industrial camera and an infrared laser line to obtain the data of the weld to be detected;

[0056] Step S5, calculating the detection result, inputting the weld data to be detected into the trained detection model to output the detection result.

[0057] Step S5 specifically includes:

[0058] Step S5-1, using the grayscale centroid method to refine the center line;

[0059] Step S5-2, for single-pass welding, the least square method LSM is used to fit the center line, and the intersection of the center line is the position of the feature point; for multi-pass welding, NURBS is used to fit the center line curve, and the position of the feature point is obtained by derivation of the fitting curve.

[0060] In step S1, in order to improve the accuracy of data acquisition, the resolution of the industrial camera is 656×492 and the wavelength of the laser transmitter is 650nm.

[0061] In step S3, the training algorithm includes a stochastic gradient training algorithm, an Adam algorithm, an RMSProp algorithm, an Adagrad algorithm, an Adadelta algorithm or an Adamax algorithm.

[0062] In step S3, for network initialization, the parameters to be trained in the network are randomly initialized using a Gaussian distribution with a standard deviation of 0.01. The learning rate and weight reduction are set to 5×10 -5 and 5×10 -4 . 3200 original weld images and weld reference images were selected, including 2400 training sample sets and 800 test sets, to train the cascade convolutional neural network. At the same time, 3200 original weld images and centerline reference images were selected, including 2400 training sample sets and 800 test sets, to train the centerline extraction network.

[0063] In step S3, the model of the cascaded convolutional neural network is as follows Figure 2 As shown. The cascade convolutional neural network includes an input layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a fifth deconvolutional layer, a sixth convolutional layer, a fourth deconvolutional layer, a seventh convolutional layer, a third deconvolutional layer, an eighth convolutional layer, a second deconvolutional layer, a first deconvolutional layer, a ninth convolutional layer, an output layer and a Softmax layer in sequence. In the continuous convolution process, a lot of feature information of the image will be lost, such as edge information, target position, etc. Weld detection requires the ability to provide accurate position information and edge information of laser stripes. Low-level convolution can extract more specific details, including laser stripe edge information, while high-level convolution can extract more position information. Therefore, the present invention adds feature fusion of low-level and high-level outputs, and supplements local detail features by performing feature fusion. In the cascade convolutional neural network, the outputs of the first convolutional layer and the second deconvolutional layer, the second convolutional layer and the third deconvolutional layer, the third convolutional layer and the fourth deconvolutional layer, and the fourth convolutional layer and the fifth deconvolutional layer are feature fused.

[0064] In step S3, in order to train the cascaded convolutional neural network, the cross entropy loss is used, and the calculation method is shown in Formula 1.

[0065]

[0066] Where θ1 represents the parameters of the cascaded convolutional neural network, as shown in Table 1, M represents the minimum batch size, N represents the number of pixels in the block, and K represents the number of categories; 1(y=k) is a function that takes 1 when y=k and 0 otherwise; represents the jth pixel in the i-th block, Indicates that the last convolutional layer is Pixel output; express The pixel to which the pixel corresponds in the true label.

[0067] Table 1 Parameters of cascaded convolutional neural network

[0068] Layer Kernel size Channels(In,Out) Stride Padding Output size Input — — — — 64×64×3 Conv1-1 3×3 3,32 2 0.5 32×32×32 Conv1-2 3×3 32,32 1 1 32×32×32 Conv2-1 3×3 32,32 2 0.5 16×16×32 Conv2-2 3×3 32,32 1 1 16×16×32 Conv3-1 3×3 32,64 2 0.5 8×8×64 Conv3-2 3×3 64,64 1 1 8×8×64 Conv4 3×3 64,128 2 0.5 4×4×128 Conv5-1 3×3 128,128 2 0.5 2×2×128 Conv5-2 3×3 128,128 1 1 2×2×128 Conv5-3 3×3 128,128 1 1 2×2×128 Conv5-4 3×3 128,128 1 1 2×2×128 DConv5 3×3 128,128 2 0.5 4×4×128 Conv6 3×3 256,128 1 1 4×4×128 DConv4 3×3 128,64 2 0.5 8×8×64 Conv7 3×3 128,64 1 1 8×8×64 DConv3 3×3 64,32 2 0.5 16×16×32 Conv8 3×3 64,32 1 1 16×16×32 DConv2 3×3 32,32 2 0.5 32×32×32 DConv1 3×3 64,32 2 0.5 64×64×32 Conv9 3×3 32,32 1 1 64×64×32 Output 3×3 32,2 1 1 64×64×2 Softmax — — — — 64×64×2

[0069] The probability that a pixel belongs to the kth class is The calculation method is shown in Formula 2.

[0070]

[0071] In step S3, the model of the centerline extraction network is as follows Figure 2As shown in the figure. The centerline extraction network includes the input layer, the first convolution layer, the first pooling layer, the second convolution layer, the second pooling layer, the third convolution layer, the third pooling layer, the third upper pooling layer, the third deconvolution layer, the second upper pooling layer, the second deconvolution layer, the first upper pooling layer, the first deconvolution layer, the output layer, and the Softmax layer. Compared with the cascaded convolutional neural network, the centerline extraction network also includes an encoding layer and a decoding layer, but the centerline extraction network structure is smaller. On the one hand, the feature map output by the last convolution layer of the cascaded convolutional neural network contains less interference information such as strong arc light, spatter, and arc noise than the original image, and the centerline extraction network can use this feature map as the input of the network. On the other hand, compared with weld detection, only fewer centerline pixels are needed to train the centerline extraction network. Overfitting often occurs in deeper networks, so choosing a smaller network is more suitable for centerline extraction.

[0072] In step S3, in order to train the centerline extraction network, the cross entropy loss is used, and the calculation method is shown in Formula 3.

[0073]

[0074] Where θ2 represents the parameters of the centerline extraction network, as shown in Table 2; 1(z=k) is a function that takes 1 when z=k and takes 0 otherwise; Represents the output of the last deconvolution layer of the center extraction network; express The pixel in the centerline that corresponds to the ground truth label.

[0075] Table 2 Parameters of centerline extraction network

[0076]

[0077]

[0078] The probability that a pixel belongs to the kth class is The calculation method is shown in Formula 4.

[0079]

[0080] In step S3, the overall network loss is calculated as shown in Formula 5.

[0081] Loss(θ1, θ2) = loss seg (y,f(x),θ1)+loss cen (z, h(x), θ2) Formula 5

[0082] In step S5-2, the NURBS curve is calculated as shown in Formula 6.

[0083]

[0084] Where P i represents the control vertex, ω i represents the weight factor, a and b are 0 and 1 respectively, N i，p (u) is the P-order B-spline basis function defined on the knot vector U.

[0085] The calculation method of the node vector U is shown in Formula 7.

[0086]

[0087] In step S5-2, single pass welding is performed as follows: Figure 3 As shown in the figure, the intersection points of the center lines are the left boundary point, the lowest point, and the right boundary point. The lowest point is the position of the next welding. Figure 4 As shown, the smallest extreme point is the position of the next welding, and the fluctuation range of welding can be limited by the minimum point and the maximum point.

[0088] The above only expresses the preferred embodiments of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that for ordinary technicians in this field, several modifications, improvements and substitutions can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. A method for automatic detection of steel welds based on cascaded convolutional neural networks, characterized in that: The following steps are involved: Step S1, obtaining weld data, scanning the steel part using an industrial camera and an infrared laser line to obtain weld data; Step S2, data annotation, annotating the weld area and center line of the image, and completing the preparation of the weld reference image set and the center line reference image set; Step S3, training the cascade convolutional neural network and the centerline extraction network, according to the training algorithm, using the weld data and the weld reference image set to train the cascade convolutional neural network, and using the weld data and the centerline reference image set to train the centerline extraction network; Step S4, obtaining the data of the weld to be detected, scanning the steel part to be detected using an industrial camera and an infrared laser line to obtain the data of the weld to be detected; Step S5, calculating the detection result, inputting the weld data to be detected into the trained detection model and outputting the detection result; In the step S3, the cascaded convolutional neural network includes an input layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a fifth deconvolutional layer, a sixth convolutional layer, a fourth deconvolutional layer, a seventh convolutional layer, a third deconvolutional layer, an eighth convolutional layer, a second deconvolutional layer, a first deconvolutional layer, a ninth convolutional layer, an output layer and a Softmax layer in sequence; low-layer convolution can extract more specific details, including laser stripe edge information, while high-layer convolution can extract more position information, and feature fusion is performed by adding the low-layer and high-layer outputs, and local detail features are supplemented by feature fusion; in the cascaded convolutional neural network, feature fusion is performed on the outputs of the first convolutional layer and the second deconvolutional layer, the second convolutional layer and the third deconvolutional layer, the third convolutional layer and the fourth deconvolutional layer, and the fourth convolutional layer and the fifth deconvolutional layer.

2. The method for automatic detection of steel welds based on a cascaded convolutional neural network according to claim 1, characterized in that: The step S2 specifically includes: Step S2-1, converting the weld data image into PNG format; Step S2-2, annotating the weld image processed by step S2-1, wherein the annotated content includes the position and size of the weld area, and the position and size of the center line.

3. The method for automatic detection of steel welds based on cascaded convolutional neural networks according to claim 1, characterized in that: In step S3, the training algorithm includes a stochastic gradient training algorithm, an Adam algorithm, an RMSProp algorithm, an Adagrad algorithm, an Adadelta algorithm or an Adamax algorithm.

4. The method for automatic detection of steel welds based on cascaded convolutional neural networks according to claim 1 is characterized in that: In step S3, in order to train the cascade convolutional neural network, a cross entropy loss is used, and the calculation method is shown in Formula 1; Where θ1 represents the parameters of the cascaded convolutional neural network, M represents the minimum batch size, N represents the number of pixels in the block, and K represents the number of categories; 1(y=k) is a function that takes 1 when y=k and 0 otherwise; represents the jth pixel in the i-th block, Indicates that the last convolutional layer is Pixel output; express The pixel to which the pixel corresponds in the true label; The probability that a pixel belongs to the kth class is The calculation method is shown in formula 2; 5. The method for automatic detection of steel welds based on cascaded convolutional neural networks according to claim 4 is characterized in that: In the step S3, the centerline extraction network includes an input layer, a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a third convolutional layer, a third pooling layer, a third upper pooling layer, a third deconvolutional layer, a second upper pooling layer, a second deconvolutional layer, a first upper pooling layer, a first deconvolutional layer, an output layer, and a Softmaxc layer in sequence; compared with the cascaded convolutional neural network, the centerline extraction network also includes an encoding layer and a decoding layer, but the centerline extraction network structure is smaller; on the one hand, the feature map output by the last convolutional layer of the cascaded convolutional neural network contains less interference information than the original image, and the centerline extraction network can use this feature map as the input of the network; on the other hand, compared with weld detection, only fewer centerline pixels are needed to train the centerline extraction network.

6. The method for automatic detection of steel welds based on cascaded convolutional neural networks according to claim 5 is characterized in that: In step S3, in order to train the centerline extraction network, a cross entropy loss is used, and the calculation method is shown in Formula 3; Where θ2 represents the parameters of the centerline extraction network, 1(z=k) is a function that takes 1 when z=k and 0 otherwise; Represents the output of the last deconvolution layer of the center extraction network; express The pixel corresponds to the pixel in the centerline ground truth label; The probability that a pixel belongs to the kth class is The calculation method is shown in formula 4; 7. The method for automatic detection of steel welds based on cascaded convolutional neural networks according to claim 6 is characterized in that: In step S3, the overall network loss is calculated as shown in Formula 5; Loss(θ1, θ2) = loss seg (y, f(x), θ1) + loss cen (z, h(x), θ2) Equation Five.

8. The method for automatic detection of steel welds based on cascaded convolutional neural networks according to claim 1, characterized in that: The step S5 specifically includes: Step S5-1, using the grayscale centroid method to refine the center line; Step S5-2, for single-pass welding, the least square method LSM is used to fit the center line, and the intersection of the center line is the position of the feature point; for multi-pass welding, NURBS is used to fit the center line curve, and the position of the feature point is obtained by derivation of the fitting curve.

9. The method for automatic detection of steel welds based on cascaded convolutional neural networks according to claim 8, characterized in that: In step S5-2, the NURBS curve is calculated as shown in Formula 6; Where P i represents the control vertex, ω i represents the weight factor, a and b are 0 and 1 respectively, N i，p (u) is the P-order B-spline basis function defined on the knot vector U; The calculation method of the node vector U is shown in Equation 7;

Images