Online real-time prediction and evaluation method and system for fillet weld penetration based on deep residual convolutional network

Abstract

The present invention provides a method and system for online, real-time prediction and assessment of fillet weld penetration based on a deep residual convolutional network. The method comprises: collecting a frontal image of the molten pool area during the fillet weld welding process; obtaining a frontal ROI image of the molten pool center using a three-class Otsu threshold segmentation method; annotating and data augmenting the ROI image of the molten pool center to obtain a penetration prediction dataset; designing a deep residual convolutional network composed of stacked residual convolutional modules that takes the molten pool image as input and outputs penetration label classification; training, validating, and testing the deep residual convolutional network; and performing real-time prediction and assessment of fillet weld penetration during the welding process based on the trained deep residual convolutional network. The present invention utilizes deep learning to achieve end-to-end real-time prediction of penetration, avoiding the cumbersome steps of traditional image processing-based feature extraction methods. This method offers improved adaptability to complex welding scenarios and high accuracy.

Description

Technical Field

[0001] The present invention relates to the field of intelligent welding technology, and in particular to a method and system for online real-time prediction and evaluation of fillet weld penetration based on a deep residual convolutional network. Background Art

[0002] During the welding process, the depth of penetration that can be achieved is an important parameter that many researchers are concerned about. How to accurately predict and control the penetration of welds in real time has always been a hot topic in the field of welding research. In traditional welding quality inspection, the weld is usually inspected after welding. The methods used include destructive inspection methods such as metallographic sectioning and mechanical property testing, non-destructive inspection methods such as appearance inspection, pressure resistance testing, sealing testing, etc., and non-destructive inspection methods such as ultrasonic testing and radiographic inspection. These inspection technologies have become more and more perfect and can basically fully reflect the quality of the weld. However, the obvious disadvantage is that the weld can only be inspected after welding is completed, and there are high technical requirements for quality inspection personnel. It is impossible to automatically obtain the welding quality status in real time during the welding process. How to achieve real-time automated inspection of weld quality is an important new topic in the field of welding quality control.

[0003] When it comes to predicting weld penetration, the most intuitive indicator of weld penetration during welding is the deepest point below the molten pool. Therefore, many researchers at home and abroad have employed various detection methods to directly detect the molten pool depth. These methods, depending on the sensing technology, can be categorized as: molten pool oscillation, ultrasonic sensing, infrared sensing, and X-ray. However, these detection methods all have their drawbacks. The molten pool oscillation method suffers from time delays, making real-time detection difficult. Ultrasonic, infrared, and X-ray detection methods also suffer from complex structures and high costs, hindering their widespread application in actual welding production.

[0004] In recent years, computer vision technology has developed rapidly and has become an important tool in the research of welding automation control. Using a camera to capture the front molten pool image and then extracting feature information from the molten pool image to reflect the penetration morphology is a common method for studying weld formation control and has been widely studied.

[0005] The invention patent with publication number CN108274095B discloses a method for detecting the welding quality of asymmetric fillet welds based on the front molten pool image features. A CCD camera is used to capture the front molten pool image of the asymmetric fillet weld. The image processing algorithm is studied to obtain the contour of the front molten pool and the position of the welding gun in the molten pool image. The relationship between the offset of the welding gun relative to the center of the molten pool and the penetration morphology is established, thereby achieving control of the welding quality.

[0006] The invention patent with publication number CN114842311A discloses a method for monitoring the welding penetration status based on multi-dimensional fusion sensing, including: collecting real-time images and real-time one-dimensional signals of the penetration status on the front side of the weld pool during welding, inputting them into a fusion model, and outputting the real-time width of the back side of the weld pool. The fusion model consists of a multi-layer perceptron neural network that inputs a one-dimensional signal and a shallow convolutional neural network that inputs a real-time image of the penetration status. The two output results are optimized using a fully connected layer to obtain the real-time width of the weld pool surface.

[0007] The invention patent with publication number CN111738369A discloses a real-time prediction method for welding penetration status and penetration depth based on the visual characteristics of the molten pool. Based on the designed molten pool visual sensing system, the two-dimensional characteristics of the molten pool are extracted in real time during the welding process, mainly the molten pool area, length and width. With these characteristics as input and the weld penetration as output, a support vector machine (SVM) and BP neural network prediction model is established.

[0008] The shortcomings of the above-mentioned penetration state prediction methods are: (1) they all use traditional image processing and then feature extraction methods to determine the penetration depth, which is cumbersome and cannot fully reflect all the characteristics of the molten pool and has poor adaptability; (2) the prediction model uses traditional machine learning methods, which has insufficient model generalization ability and is difficult to adapt to the complex and diverse weld forms on site; (3) they are mostly used for post-weld inspection and do not have the function of real-time prediction of penetration depth during welding and the function of evaluating the penetration state throughout the welding process. Summary of the Invention

[0009] In view of the defects in the prior art, the purpose of the present invention is to provide a method and system for online real-time prediction and evaluation of fillet weld penetration based on deep residual convolutional network.

[0010] The method for online real-time prediction and evaluation of fillet weld penetration based on a deep residual convolutional network provided by the present invention includes:

[0011] Step S1: Collect the front image of the molten pool area during the fillet weld process. Fix the CCD camera on the welding robot, design and carry out the orthogonal experiment of welding process parameters, collect images, and perform preliminary screening on the collected images.

[0012] Step S2: Based on the characteristics of the collected front image of the molten pool area, a three-category Otsu threshold segmentation method is used to obtain a molten pool center ROI image of the front molten pool image;

[0013] Step S3: Annotate and augment the ROI image of the molten pool center to obtain a fillet weld penetration prediction dataset;

[0014] Step S4: Based on the obtained fillet weld penetration prediction dataset, a deep residual convolutional network consisting of a stack of residual convolution modules is designed, which takes the molten pool image as input and the penetration label classification as output. The deep residual convolutional network is then trained, verified, and tested.

[0015] Step S5: Based on the trained deep residual convolutional network, the fillet weld penetration is predicted in real time during the welding process.

[0016] Preferably, the step S1 includes:

[0017] Step S1.1: Install and debug the experimental device: Make the center of the welding gun perpendicular to the horizontal plane; install the CCD camera on the welding robot along the welding direction, adjust and fix the CCD camera posture so that the center of the welding gun is located in the center of the CCD camera's field of view; place the workpiece on the bracket;

[0018] Step S1.2: Welding experiment design: Using orthogonal experimental design, set up a three-factor multi-level experiment for wire feed speed, welding gun angle, and arc length correction factor; conduct welding experiments for each parameter group;

[0019] Step S1.3: Acquire the molten pool image during the welding process at a fixed frame rate, and filter the molten pool image according to a preset image quality condition.

[0020] Preferably, step S2 includes:

[0021] Step S2.1: First, filter the image to remove the noise in the molten pool image and reduce the interference of arc light on image processing;

[0022] Step S2.2: Using histogram equalization to automatically adjust the grayscale transformation to expand the dynamic range of the image and improve the contrast and grayscale tone changes;

[0023] Step S2.3: Threshold processing is performed using the three-class Otsu threshold method to obtain the ROI image of the melt pool center;

[0024] Step S2.4: Screen the melt pool ROI and remove ROI images that do not meet the requirements in size and grayscale distribution.

[0025] Preferably, step S3 includes:

[0026] Step S3.1: Metallographic observation to determine the corresponding penetration depth of the experimental parameters: Samples are cut perpendicular to the weld axis from the front, middle, and back sections of the fillet weld specimens after welding. After cutting, grinding, and polishing, the specimens are etched with an etchant. The penetration depths of both sides of the fillet welds in different groups are observed under a metallographic microscope. The penetration depths of both sides of the fillet welds in different groups are recorded. The average of the penetration depths at the front, middle, and back sections is used as the penetration depth of the weld.

[0027] Step S3.2: Dataset construction: The ROI image of the center of the weld pool and the corresponding weld penetration label in each group of experiments constitute the fillet weld penetration prediction dataset;

[0028] Step S3.3: Dataset division: The obtained dataset is divided into a training set, a validation set, and a test set according to a preset ratio.

[0029] Preferably, step S4 includes:

[0030] Step S4.1: Design a deep residual convolutional network consisting of a stack of residual convolutional modules, which takes the melt pool image as input and outputs the melt depth label classification;

[0031] Step S4.2: Configure the training method, including setting the learning rate, decay method, loss function, batch size, and number of iterations.

[0032] Step S4.3: Preprocess the melt pool center ROI image: First, convert the grayscale image into a three-channel color image, then compress the image and crop it to the standard input size of the deep residual convolutional network. The preprocessed image is used as the input of the deep residual convolutional network.

[0033] Step S4.4: Use the pre-trained weights of the deep residual convolutional network for transfer learning, freeze the weights of each convolutional layer, and only train the weights of the fully connected layers related to the specific classification;

[0034] Step S4.5: Use the validation set to perform an overfitting check to determine whether the loss function of the deep residual learning network for the validation set images first decreases and then increases. Stop training when the loss function of the validation set no longer decreases.

[0035] Step S4.6: Test the accuracy and reliability of the deep residual recognition model using the test set.

[0036] The online real-time prediction and evaluation system for fillet weld penetration based on a deep residual convolutional network provided by the present invention includes:

[0037] Image acquisition module M1: collects frontal images of the molten pool during fillet weld welding. The CCD camera is fixed to the welding robot, and an orthogonal experiment of welding process parameters is designed and carried out to acquire images and perform preliminary screening of the acquired images.

[0038] ROI extraction module M2: Based on the characteristics of the collected front image of the melt pool area, the three-category Otsu threshold segmentation method is used to obtain the melt pool center ROI image of the front melt pool image;

[0039] Dataset construction module M3: annotates and augments the ROI image of the molten pool center to obtain a fillet weld penetration prediction dataset;

[0040] Network construction module M4: Based on the obtained fillet weld penetration prediction dataset, a deep residual convolutional network consisting of stacked residual convolution modules is designed, which takes the weld pool image as input and outputs the penetration label classification. The deep residual convolutional network is then trained, verified, and tested.

[0041] Real-time prediction module M5: Based on the trained deep residual convolutional network, the fillet weld penetration depth is predicted in real time during the welding process.

[0042] Preferably, the module M1 includes:

[0043] Module M1.1: Install and debug the experimental device: Make the center of the welding gun perpendicular to the horizontal plane; install the CCD camera on the welding robot along the welding direction, adjust and fix the CCD camera posture so that the center of the welding gun is in the center of the CCD camera's field of view; place the workpiece on the bracket;

[0044] Module M1.2: Welding Experiment Design: Using orthogonal experimental design, set up a three-factor multi-level experiment for wire feed speed, welding gun angle, and arc length correction factor; conduct welding experiments for each parameter group;

[0045] Module M1.3: Acquire the molten pool image during the welding process at a fixed frame rate and filter the molten pool image according to the preset image quality conditions.

[0046] Preferably, the module M2 includes:

[0047] Module M2.1: First, filter the image to remove the noise in the molten pool image and reduce the interference of arc light on image processing;

[0048] Module M2.2: Utilizes grayscale transformation through histogram equalization to automatically adjust and expand the dynamic range of the image, improving contrast and grayscale tone changes;

[0049] Module M2.3: Use the three-class Otsu threshold method to perform threshold processing and obtain the ROI image of the melt pool center;

[0050] Module M2.4: Screen the melt pool ROI and remove ROI images whose size and grayscale distribution do not meet the requirements.

[0051] Preferably, the module M3 includes:

[0052] Module M3.1: Metallographic Observation to Determine the Corresponding Penetration Depth of Experimental Parameters: For fillet weld specimens, take the front, middle, and back sections of the weld specimen after welding. Cut the specimens perpendicular to the weld axis. After cutting, grinding, and polishing, etch them with an etchant. Observe the penetration of both sides of different groups of fillet welds under a metallographic microscope. Record the penetration depth of both sides of different groups of fillet welds. The average of the penetration depths before, during, and after welding is used as the penetration depth of the weld.

[0053] Module M3.2: Dataset construction: The ROI image of the center of the weld pool and the corresponding weld penetration label in each group of experiments constitute the fillet weld penetration prediction dataset;

[0054] Module M3.3: Dataset division: Divide the obtained dataset into training set, validation set and test set according to the preset ratio.

[0055] Preferably, the module M4 includes:

[0056] Module M4.1: Design a deep residual convolutional network consisting of stacked residual convolutional modules, taking melt pool images as input and outputting melt depth label classifications;

[0057] Module M4.2: Configure the training method, including setting the learning rate, decay method, loss function, batch size, and number of iterations.

[0058] Module M4.3: Image preprocessing of the melt pool center ROI image: First, convert the grayscale image into a three-channel color image, then compress the image and crop it to the standard input size of the deep residual convolutional network. The preprocessed image is used as the input of the deep residual convolutional network.

[0059] Module M4.4: Use pre-trained weights of deep residual convolutional networks for transfer learning, freezing the weights of each convolutional layer and training only the weights of the fully connected layers related to specific classifications;

[0060] Module M4.5: Use the validation set to perform an overfitting check to determine whether the loss function of the deep residual learning network for the validation set images first decreases and then increases. Stop training when the loss function of the validation set no longer decreases.

[0061] Module M4.6: Test the accuracy and reliability of the deep residual recognition model using a test set.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] (1) The present invention uses three types of OTSU threshold segmentation algorithms to extract the ROI of the molten pool center based on the illumination characteristics of the molten pool, reducing the interference of welding background and allowing the deep learning model to focus on the molten pool characteristics themselves;

[0064] (2) Traditional penetration prediction mostly uses multi-step image processing algorithms. On the one hand, the processing algorithm and feature extraction rely on professional research. On the other hand, the feature extraction algorithm is not suitable for dynamic changes in illumination and molten pool. The present invention adopts deep learning and makes full use of its feature extraction capability to achieve end-to-end penetration prediction, avoiding the cumbersome steps of traditional feature extraction methods based on image processing, and has better adaptability to complex welding scenarios and high accuracy.

[0065] (3) The deep residual convolutional network used in the present invention adopts a residual structure to avoid the gradient vanishing and gradient exploding problems during the training process, making the model easier to train while deepening the model depth;

[0066] (4) Traditional penetration depth prediction is mostly used for post-weld inspection. The present invention applies the trained model to the penetration depth prediction of the on-site real-time welding process, thereby realizing the real-time prediction and evaluation of the penetration depth of the fillet weld during the welding process. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0068] Figure 1 This is a flowchart of the online real-time prediction and evaluation method for fillet weld penetration based on deep residual convolutional network;

[0069] Figure 2 Schematic diagram of the prediction of fillet weld penetration based on frontal weld pool vision;

[0070] Figure 3 Orthogonal design table for welding experiment;

[0071] Figure 4a and Figure 4b This is the ROI extraction result of the front melt pool center;

[0072] Figure 5 This is the metallographic observation diagram of the fillet weld penetration;

[0073] Figure 6 Classify the front melt pool center ROI image dataset;

[0074] Figure 7a to Figure 7c It is a deep residual convolutional network structure;

[0075] Figure 8 Convergence of the loss function for training a deep residual convolutional network for the dataset;

[0076] Figure 9a and Figure 9b The prediction effect of fillet weld penetration prediction using the test set is shown;

[0077] Figure 10 A real-time predicted curve of penetration fluctuation of a single weld.

[0078] Figure 11 This is a statistical diagram of the penetration depth of a single weld after welding is completed. DETAILED DESCRIPTION

[0079] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0080] Example 1

[0081] The present invention provides an online real-time prediction and evaluation method for fillet weld penetration based on a deep residual convolutional network. The method uses a 50-layer deep residual convolutional network to automatically identify the penetration of the fillet weld molten pool center ROI image. The specific details are as follows:

[0082] A CCD camera is used to capture the front image of the molten pool area during the welding process. First, the front image of the molten pool area is preprocessed to obtain the molten pool center ROI image, which is then filtered and input into the 50-layer depth residual fillet weld penetration prediction model, thereby realizing real-time penetration prediction of the fillet weld during welding. Figure 1 , specifically including the following steps:

[0083] Step S1: Capture a frontal image of the molten pool during fillet welding. A CCD camera is mounted on a welding robot. An orthogonal experiment of welding process parameters is designed and conducted to capture images. The captured images are then preliminarily screened.

[0084] The step 1 comprises:

[0085] Step S1.1: Install and debug the experimental device. Make the center of the welding gun perpendicular to the horizontal plane; install the CCD camera on the welding robot along the welding direction, adjust and fix the CCD camera posture so that the center of the welding gun is located in the center of the CCD camera field of view; place the workpiece on the bracket; Figure 2 As shown, the CCD camera used is a Xiris XVC-1000 welding industrial camera, the filter is a narrow-band filter with a central wavelength of 650 nm, and the robot is a Kuka robot.

[0086] Step S1.2: Welding Experiment Design. Using an orthogonal experimental design, a three-factor, multi-level experiment was set up for wire feed speed, welding gun angle, and arc length correction factor. Welding experiments were conducted for each parameter group. The test material was B980LE steel. The fillet weld specimen dimensions were 400 mm (length) × 90 mm (width) × 35 mm (height), with an 8 mm bottom plate thickness and a 4.5 mm side plate thickness. Double-sided fillet welds were welded without a groove, and each weld was approximately 100 mm long. The wire feeding speed levels are 5.2mm / s, 5.6mm / s, 6mm / s, 6.4mm / s, 6.8mm / s, 7.2mm / s, 7.6mm / s, 8mm / s, 8.4mm / s, 8.8mm / s, 9.2mm / s, and 9.6mm / s. The welding gun angle is the actual angle between the welding gun and one side of the base plate, which is 35°, 40°, 48°, 52°, 56°, and 44°. The arc length correction coefficient is 0, 4, 8, and 10. There are 72 groups of experiments in total. Figure 3 shown.

[0087] Step S1.3: Acquire images of the molten pool during the welding process at a fixed frame rate, removing images where the arc light interferes severely and the molten pool is difficult to identify. The frame rate is set to 25 fps.

[0088] Step S2: Based on the characteristics of the collected front image of the molten pool area, the three-category Otsu threshold segmentation method is used to obtain the front molten pool center ROI image.

[0089] The step S2 comprises:

[0090] Step S2.1: First, perform median filtering and Butterworth low-pass filtering on the image to remove noise in the molten pool image and reduce the interference of arc light on image processing;

[0091] Step S2.2: Using histogram equalization to automatically adjust the grayscale transformation to expand the dynamic range of the image and improve the contrast and grayscale tone changes;

[0092] Step S2.3: Use the three-class Otsu threshold method to perform threshold processing to obtain the ROI image of the center of the melt pool.

[0093] In step S2.3, the three-class Otsu method applies the Otsu method to three-class segmentation, denoted as C1, C2, and C3, and the inter-class variance of the image is:

[0094]

[0095] in,

[0096]

[0097]

[0098] At the same time:

[0099] P1m1+P2m2+P3m3=m G ,P1+P2+P3=1

[0100] The goal of the algorithm is to maximize the inter-class variance of the three categories calculated according to the threshold:

[0101]

[0102] The specific algorithm steps are:

[0103] 1. Initialize k1 = 1;

[0104] 2. Traverse k2, whose value range is [k1, L-1];

[0105] 3. Let k1 = k1 + 1, repeat step 2 above, calculate and record Until k1=L-3, the traversal ends;

[0106] 4. Find the When the maximum and That is the segmentation threshold of the melt pool ROI;

[0107] 5. Segment the image: Divide the image into three parts: background area, transition area, and melt pool ROI according to the following formula, thereby obtaining the melt pool center ROI.

[0108]

[0109] Where a, b, and c are the grayscale thresholds of the background area, transition area, and melt pool ROI area, respectively.

[0110] Assuming a = 0, b = 128, and c = 255, the result of threshold segmentation is shown in Figure 4. The part c = 255 is the area where the molten pool ROI is located.

[0111] Step S2.4: Screen the melt pool ROI and remove ROI images that do not meet the requirements in size and grayscale distribution.

[0112] Based on the actual weld size and parameter variation range, determine the appropriate ROI length, width, and aspect ratio thresholds, as well as the grayscale distribution threshold for the normal weld pool image ROI area, and filter the weld pool image. Only those that meet the following three requirements are set as qualified ROIs.

[0113] (a) The length and width of the ROI area are within the preset threshold;

[0114] (b) The aspect ratio of the ROI area is within the preset threshold;

[0115] (c) The grayscale value change of the image within the ROI is within the preset threshold.

[0116] Step S3: annotate and augment the filtered molten pool center ROI image to obtain a fillet weld penetration prediction dataset;

[0117] The step S3 comprises:

[0118] Step S3.1: Metallographic observation to determine the corresponding penetration depth of the experimental parameters. Take the front, middle and back sections of the fillet weld specimen after welding, cut the specimens perpendicular to the weld axis, cut, grind and polish them, and then etch them with an etchant. Observe the penetration of both sides of the fillet welds of different groups under a metallographic microscope. Figure 5 As shown in the figure, the penetration depths on both sides of different groups of fillet welds are recorded, and the average penetration depth before, during and after the weld is taken as the penetration depth of the weld.

[0119] The penetration label is divided into eight categories based on the ratio of the maximum side plate penetration to the side plate thickness. Using 0.1 as the boundary (considering that penetrations of 0-0.1 and 0.9-1.0 are less common in actual welding, 0-0.2 and 0.8-1.0 are set as one category), the penetration is divided into 0-0.2, 0.2-0.3, 0.3-0.4, 0.4-0.5, 0.5-0.6, 0.6-0.7, 0.7-0.8, and 0.8-1.0, totaling 8 categories.

[0120] Step S3.2: The ROI image of the molten pool center and the corresponding weld penetration label in each group of experiments constitute the fillet weld penetration prediction dataset;

[0121] Step S3.3: The obtained dataset is classified into training set, validation set and test set according to the ratio of 80%, 10% and 10%, and the final training set images are 19067, the validation set images are 2383 and the test set images are 2380. Figure 6 shown.

[0122] Step S4: Based on the obtained fillet weld penetration prediction dataset, a deep residual convolutional network consisting of stacked residual convolution modules is designed, which takes the molten pool image as input and the penetration label classification as output. The deep residual convolutional network is trained, verified and tested.

[0123] The step S4 specifically includes:

[0124] Step S4.1: Design a deep residual convolutional network consisting of a stack of residual convolutional modules, with the melt pool image as input and the melt depth label classification as output:

[0125] There are two types of residual convolution modules: convolutional modules and residual modules;

[0126] The structure of the convolution module (Conv Block) is divided into two parts, such as Figure 7a As shown in the figure. The left side is the main part, which has two convolutions (1×1, 3×3), normalization, activation function (ReLU) and one convolution (1×1), normalization; the right side is the residual part, which has one convolution (1×1) and normalization. Since the output and output dimensions of the convolution module are different, they cannot be connected in series. The purpose is to change the dimension of the network.

[0127] The structure of the residual module (Identity Block) is divided into two parts, such as Figure 7b As shown in the figure. The left side is the main part, which has two convolutions (1×1, 3×3), normalization, activation function (ReLU) and one convolution (1×1), normalization; the right side is the residual part, which has no convolution operation; since the input feature layer dimension of the residual module is the same as the output feature layer, they can be connected in series to deepen the network;

[0128] The entire network structure consists of three parts: input layer, residual learning layer and fully connected output layer, such as Figure 7c shown.

[0129] The input layer conv1 consists of a zero-padding layer (with a stride of 3), a 64-channel convolution of [1,1], normalization, activation function (ReLU) and maximum pooling (3×3); the input image size is [256,256,3], and a preliminary effective feature layer of [112,112,64] is obtained after the input layer, followed by a 3×3 maximum pooling to obtain a feature layer of [56,56,64].

[0130] The residual learning layer consists of four groups: conv2, conv3, conv4, and conv5. Each group consists of a convolutional module plus several residual modules. The number of residual modules in conv2 is 2, the number of residual modules in conv3 is 3, the number of residual modules in conv4 is 5, and the number of residual modules in conv5 is 2. The stride of the first convolutional layer of the convolutional module in conv2 and the convolutional layer on the residual module is 1, while the stride of the first convolutional layer of the convolutional module in conv3, conv4, and conv5 and the convolutional layer on the residual module is 2. After conv2, a feature layer of [56, 56, 256] is obtained, after conv3, a feature layer of [28, 28, 512] is obtained, after conv4, a feature layer of [14, 14, 1024] is obtained, and after conv5, a feature layer of [7, 7, 2048] is obtained.

[0131] The output layer uses a 7×7 average pooling to obtain the [1,1,2048] feature layer, which is flattened and connected to a 1×8 fully connected layer. The Softmax activation function is used to output the classification result of the fillet weld penetration and the corresponding probability Y i (Y i The output result indicates that the image belongs to the i-th category, where i = (1, 2, 3, 4, 5, 6, 7, 8). The mathematical expression of the probability Yi is as follows:

[0132]

[0133] Among them, Y i represents the output result of the i-th penetration category, K represents the number of categories, and e represents the natural logarithm.

[0134] The loss function used by the classification network is the cross entropy loss function (Cross Entropy Loss), and the calculation formula is as follows:

[0135]

[0136] Where M is the number of categories, y ic is the true label (0 or 1), when the i-th sample belongs to class c, the value is 1, otherwise it is 0; p ic is the prediction result, the predicted probability that the i-th sample belongs to class c; i represents the number of samples.

[0137] Step S4.2: Configure the training method, including setting the learning rate, decay method, loss function, batch size, and number of iterations.

[0138] The Adam (Adaptive moment estimation) optimization algorithm is used to update the model parameters. The initial learning rate is set to 1e-2, and the minimum learning rate of the model is set to 1e-4. The learning rate decay method is used, the decay momentum is 0.9, the batch_size is 32, and the number of iterations is 50.

[0139] Step S4.3: Perform image preprocessing on the melt pool center ROI image. First, convert the grayscale image into a three-channel color image. Then, compress the image and crop it to the standard input size of the deep residual convolutional network. The preprocessed image is used as the input of the deep residual convolutional network.

[0140] The input image is cropped and compressed to a size of 224×224, and all the cropped and compressed images are put into the deep residual convolutional network as input as a dataset;

[0141] Step S4.4: Use the pre-trained weights of the deep residual convolutional network for transfer learning. Freeze the weights of the convolutional layers conv1, conv2, conv3, conv4, and conv5, and only train the weights of the fully connected layers related to the specific classification.

[0142] Step S4.5: Use the validation set to perform an overfitting check. Determine whether the loss function of the deep residual learning network for the validation set images first decreases and then increases. Stop training when the loss function of the validation set no longer decreases.

[0143] The convergence of the loss function of the residual convolutional network for predicting fillet weld penetration depth on the training set and validation set during the training process is shown in the figure below. Figure 8 shown.

[0144] Step S4.6: Test the accuracy and reliability of the deep residual recognition model using the test set.

[0145] As shown in Figure 9, the prediction accuracy and recall rate of eight categories of fillet weld penetration obtained from the test set are shown.

[0146] Step S5: Based on the trained deep residual convolutional network, the fillet weld penetration is predicted in real time during the welding process.

[0147] Step S5 includes: during the fillet weld welding process, the CCD camera simultaneously captures the weld pool image and inputs it into the trained deep residual convolutional network for discrimination, that is, it can realize real-time detection and evaluation of the fillet weld penetration during the welding process, providing a basis for welding quality control and process optimization.

[0148] Specific penetration assessments include:

[0149] Degree of penetration: Based on the predicted penetration depth, the degree of penetration is divided into four categories: insufficient penetration, normal penetration, excessive penetration and weld leakage. That is, a penetration depth of 0-0.3 indicates insufficient penetration, 0.3-0.7 indicates normal penetration, 0.7-0.9 indicates excessive penetration, and 0.9-1.0 indicates weld leakage.

[0150] Penetration fluctuation: Calculate the penetration fluctuation statistics of a single weld, including mean, standard deviation, and variance, and present them in a curve graph, such as Figure 10 As shown;

[0151] Penetration statistics: Get the penetration distribution of a single weld, including penetration position, quantity, length and percentage, and present it in a pie chart, such as Figure 11 As shown;

[0152] Out-of-tolerance statistics: Calculates the distribution of penetration depths exceeding a given standard value, including penetration position, quantity, length, and percentage.

[0153] Example 2

[0154] The present invention also provides an online real-time prediction and evaluation system for fillet weld penetration based on a deep residual convolutional network. The online real-time prediction and evaluation system for fillet weld penetration based on a deep residual convolutional network can be implemented by executing the process steps of the online real-time prediction and evaluation method for fillet weld penetration based on a deep residual convolutional network. That is, those skilled in the art can understand the online real-time prediction and evaluation method for fillet weld penetration based on a deep residual convolutional network as an optimal implementation of the online real-time prediction and evaluation system for fillet weld penetration based on a deep residual convolutional network.

[0155] The online real-time prediction and evaluation system for fillet weld penetration based on a deep residual convolutional network provided by the present invention includes: an image acquisition module M1: which acquires a front image of the molten pool area during the fillet weld welding process, fixes a CCD camera on a welding robot, designs and conducts an orthogonal experiment of welding process parameters, performs image acquisition, and performs preliminary screening of the acquired images; a ROI extraction module M2: which uses a three-category Otsu threshold segmentation method based on the characteristics of the acquired front image of the molten pool area to obtain a molten pool center ROI image of the front molten pool image; a data set construction module M3: which labels and augments the molten pool center ROI image to obtain a fillet weld penetration prediction data set; a network construction module M4: which designs a deep residual convolutional network composed of stacked residual convolution modules, with the molten pool image as input and the penetration label classification as output, based on the obtained fillet weld penetration prediction data set, and trains, verifies, and tests the deep residual convolutional network; and a real-time prediction module M5: which performs real-time prediction of fillet weld penetration during the welding process based on the trained deep residual convolutional network.

[0156] The module M1 includes: module M1.1: installation and debugging of experimental equipment: making the center of the welding gun perpendicular to the horizontal plane; installing the CCD camera on the welding robot along the welding direction, adjusting and fixing the CCD camera posture so that the center of the welding gun is located at the center of the CCD camera field of view; placing the workpiece on the bracket; module M1.2: welding experiment design: using orthogonal experimental design, setting up a three-factor multi-level experiment for wire feeding speed, welding gun angle and arc length correction coefficient; conducting welding experiments for each group of parameters; module M1.3: acquiring the molten pool image during the welding process at a fixed frame rate, and screening the molten pool image according to preset image quality conditions.

[0157] The module M2 includes: module M2.1: first filtering the image to remove the noise in the molten pool image and reduce the interference of arc light on image processing; module M2.2: using grayscale transformation through histogram equalization to automatically adjust and expand the dynamic range of the image, improve the contrast and grayscale tone changes; module M2.3: using the three-category Otsu threshold method to perform threshold processing to obtain the molten pool center ROI image; module M2.4: screening the molten pool ROI and eliminating ROI images whose size and grayscale distribution do not meet the requirements.

[0158] The module M3 includes: module M3.1: metallographic observation, determining the corresponding penetration depth of experimental parameters: taking the front, middle and rear sections of the fillet weld specimen after welding, cutting the specimen in a direction perpendicular to the weld axis, cutting, grinding and polishing, and then corroding with a corrosive agent, observing the penetration conditions on both sides of different groups of fillet welds under a metallographic microscope, recording the penetration depths on both sides of different groups of fillet welds, and taking the average of the penetration depths before, during and after the welding as the penetration depth of the weld; module M3.2: data set construction: the ROI image of the center of the molten pool and the penetration depth label of the corresponding weld in each group of experiments constitute the fillet weld penetration prediction data set; module M3.3: data set division: the obtained data set is divided into a training set, a validation set and a test set according to a preset ratio.

[0159] The module M4 includes: module M4.1: designing a deep residual convolutional network composed of stacked residual convolution modules with the melt pool image as input and the melt depth label classification as output; module M4.2: configuring the training method, including setting the learning rate, attenuation method, loss function, batch size batch_size, and number of iterations epoch; module M4.3: performing image preprocessing on the melt pool center ROI image: first converting the grayscale image into a three-channel color image, then compressing the image and cropping it to the standard input size of the deep residual convolutional network, and pre-processing the image. The processed image is used as the input of the deep residual convolutional network; Module M4.4: Use the pre-trained weights of the deep residual convolutional network for transfer learning, freeze the weights of each convolutional layer, and only train the weights of the fully connected layer related to the specific classification; Module M4.5: Use the validation set to check for overfitting to determine whether the loss function of the deep residual learning network for the validation set image processing shows a phenomenon of first decreasing and then increasing. When the loss function of the validation set no longer decreases, stop training; Module M4.6: Test the accuracy and reliability of the deep residual recognition model through the test set.

[0160] Those skilled in the art will appreciate that, in addition to implementing the system, device, and various modules provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; the modules for implementing various functions can also be considered both software programs for implementing the method and structures within the hardware component.

[0161] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A method for online real-time prediction and evaluation of fillet weld penetration based on deep residual convolutional network, characterized in that: include: Step S1: Collect the front image of the molten pool area during the fillet weld process. Fix the CCD camera on the welding robot, design and carry out the orthogonal experiment of welding process parameters, collect images, and perform preliminary screening on the collected images. Step S2: Based on the characteristics of the collected front image of the molten pool area, a three-category Otsu threshold segmentation method is used to obtain a molten pool center ROI image of the front molten pool image; Step S3: Annotate and augment the ROI image of the molten pool center to obtain a fillet weld penetration prediction dataset; Step S4: Based on the obtained fillet weld penetration prediction dataset, a deep residual convolutional network consisting of a stack of residual convolution modules is designed, which takes the molten pool image as input and the penetration label classification as output. The deep residual convolutional network is then trained, verified, and tested. Step S5: Based on the trained deep residual convolutional network, the fillet weld penetration is predicted in real time during the welding process; The step S1 comprises: Step S1.1: Install and debug the experimental device: Make the center of the welding gun perpendicular to the horizontal plane; install the CCD camera on the welding robot along the welding direction, adjust and fix the CCD camera posture so that the center of the welding gun is located in the center of the CCD camera's field of view; place the workpiece on the bracket; Step S1.2: Welding experiment design: Using orthogonal experimental design, set up a three-factor multi-level experiment for wire feed speed, welding gun angle, and arc length correction factor; conduct welding experiments for each parameter group; Step S1.3: Acquire the molten pool image during the welding process at a fixed frame rate, and filter the molten pool image according to a preset image quality condition; The step S2 comprises: Step S2.1: First, filter the image to remove the noise in the molten pool image and reduce the interference of arc light on image processing; Step S2.2: Using histogram equalization to automatically adjust the grayscale transformation to expand the dynamic range of the image and improve the contrast and grayscale tone changes; Step S2.3: Threshold processing is performed using the three-class Otsu threshold method to obtain the ROI image of the melt pool center; Step S2.4: Screen the melt pool ROI and remove ROI images that do not meet the requirements in size and grayscale distribution; The step S3 comprises: Step S3.1: Metallographic observation to determine the corresponding penetration depth of the experimental parameters: Samples are cut perpendicular to the weld axis from the front, middle, and back sections of the fillet weld specimens after welding. After cutting, grinding, and polishing, the specimens are etched with an etchant. The penetration depths of both sides of the fillet welds in different groups are observed under a metallographic microscope. The penetration depths of both sides of the fillet welds in different groups are recorded. The average of the penetration depths at the front, middle, and back sections is used as the penetration depth of the weld. Step S3.2: Dataset construction: The ROI image of the molten pool center and the corresponding weld penetration label in each group of experiments constitute the fillet weld penetration prediction dataset; Step S3.3: Dataset division: Divide the obtained dataset into training set, validation set and test set according to the preset ratio; The step S4 comprises: Step S4.1: Design a deep residual convolutional network consisting of a stack of residual convolutional modules, which takes the melt pool image as input and outputs the melt depth label classification; Step S4.2: Configure the training method, including setting the learning rate, decay method, loss function, batch size, and number of iterations. Step S4.3: Preprocess the melt pool center ROI image: First, convert the grayscale image into a three-channel color image, then compress the image and crop it to the standard input size of the deep residual convolutional network. The preprocessed image is used as the input of the deep residual convolutional network. Step S4.4: Use the pre-trained weights of the deep residual convolutional network for transfer learning, freeze the weights of each convolutional layer, and only train the weights of the fully connected layers related to the specific classification; Step S4.5: Use the validation set to perform an overfitting check to determine whether the loss function of the deep residual learning network for the validation set images first decreases and then increases. Stop training when the loss function of the validation set no longer decreases. Step S4.6: Test the accuracy and reliability of the deep residual recognition model using the test set.

2. An online real-time prediction and evaluation system for fillet weld penetration based on deep residual convolutional network, characterized in that: include: Image acquisition module M1: collects frontal images of the molten pool during fillet weld welding. The CCD camera is fixed to the welding robot, and an orthogonal experiment of welding process parameters is designed and carried out to acquire images and perform preliminary screening of the acquired images. ROI extraction module M2: Based on the characteristics of the collected front image of the melt pool area, the three-category Otsu threshold segmentation method is used to obtain the melt pool center ROI image of the front melt pool image; Dataset construction module M3: annotates and augments the ROI image of the molten pool center to obtain a fillet weld penetration prediction dataset; Network construction module M4: Based on the obtained fillet weld penetration prediction dataset, a deep residual convolutional network consisting of stacked residual convolution modules is designed, which takes the weld pool image as input and outputs the penetration label classification. The deep residual convolutional network is then trained, verified, and tested. Real-time prediction module M5: Based on the trained deep residual convolutional network, it performs real-time prediction of fillet weld penetration during the welding process; The module M1 includes: Module M1.1: Install and debug the experimental device: Make the center of the welding gun perpendicular to the horizontal plane; install the CCD camera on the welding robot along the welding direction, adjust and fix the CCD camera posture so that the center of the welding gun is in the center of the CCD camera's field of view; place the workpiece on the bracket; Module M1.2: Welding Experiment Design: Using orthogonal experimental design, set up a three-factor multi-level experiment for wire feed speed, welding gun angle, and arc length correction factor; conduct welding experiments for each parameter group; Module M1.3: Acquire the molten pool image during the welding process at a fixed frame rate and filter the molten pool image according to the preset image quality conditions; The module M2 includes: Module M2.1: First, filter the image to remove the noise in the molten pool image and reduce the interference of arc light on image processing; Module M2.2: Utilizes grayscale transformation through histogram equalization to automatically adjust and expand the dynamic range of the image, improving contrast and grayscale tone changes; Module M2.3: Use the three-class Otsu threshold method to perform threshold processing and obtain the ROI image of the melt pool center; Module M2.4: Screen the melt pool ROI and remove ROI images that do not meet the requirements in size and grayscale distribution; The module M3 includes: Module M3.1: Metallographic Observation to Determine the Corresponding Penetration Depth of Experimental Parameters: For fillet weld specimens, take the front, middle, and back sections of the weld specimen after welding. Cut the specimens perpendicular to the weld axis. After cutting, grinding, and polishing, etch them with an etchant. Observe the penetration of both sides of different groups of fillet welds under a metallographic microscope. Record the penetration depth of both sides of different groups of fillet welds. The average of the penetration depths before, during, and after welding is used as the penetration depth of the weld. Module M3.2: Dataset construction: The ROI image of the center of the weld pool and the corresponding weld penetration label in each group of experiments constitute the fillet weld penetration prediction dataset; Module M3.3: Dataset division: Divide the obtained dataset into training set, validation set and test set according to the preset ratio; The module M4 includes: Module M4.1: Design a deep residual convolutional network consisting of stacked residual convolutional modules, taking melt pool images as input and outputting melt depth label classifications; Module M4.2: Configure the training method, including setting the learning rate, decay method, loss function, batch size, and number of iterations. Module M4.3: Image preprocessing of the melt pool center ROI image: First, convert the grayscale image into a three-channel color image, then compress the image and crop it to the standard input size of the deep residual convolutional network. The preprocessed image is used as the input of the deep residual convolutional network. Module M4.4: Use pre-trained weights of deep residual convolutional networks for transfer learning, freezing the weights of each convolutional layer and training only the weights of the fully connected layers related to specific classifications; Module M4.5: Use the validation set to perform an overfitting check to determine whether the loss function of the deep residual learning network for the validation set images first decreases and then increases. Stop training when the loss function of the validation set no longer decreases. Module M4.6: Test the accuracy and reliability of the deep residual recognition model using a test set.

Images