Visual monitoring method of welding molten pool based on multimodal semi-supervised semantic segmentation

Abstract

The present invention belongs to the field of image processing technology, and discloses a welding pool visual monitoring method based on multimodal semi-supervised semantic segmentation, firstly constructing a semantic segmentation network; training the semantic segmentation network based on a semi-supervised semantic segmentation network training strategy; inputting the collected original image of the welding pool into the trained semantic segmentation network to obtain a segmented welding pool image. The present invention utilizes a multimodal semi-supervised learning strategy combining supervised learning based on a small amount of labeled data and unsupervised learning of pseudo-labeled data of different modalities to achieve effective training of the semantic segmentation network, while improving the training efficiency of the semantic segmentation network, ensuring the accuracy of welding pool detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and relates to electric welding monitoring based on image processing, and in particular to a welding molten pool visual monitoring method based on multimodal semi-supervised semantic segmentation. Background Art

[0002] In the arc welding process, real-time response to the welding status is of great significance for adjusting the welding process parameters. An important way is to dynamically monitor the molten pool generated during the welding process. With the development of machine vision technology, the use of imaging methods to observe the molten pool morphology and reflect the welding status has become a research hotspot in the field of intelligent welding. At present, the work on visual detection of welding molten pools can be divided into two categories: welding monitoring methods based on image processing and welding monitoring methods based on deep learning.

[0003] The welding monitoring method based on image processing is a multi-stage method, which mainly includes grayscale transformation and morphological processing. Xu Zihui et al. disclosed the analysis of molten pool images based on the change of robot welding gun inclination angle. The collected molten pool images were preprocessed, morphologically processed and Canny operator was used to extract edges to obtain a molten pool image with smooth edge contours, and its area and aspect ratio were calculated [Xu Zihui, Yao Ping, Liang Daozan, et al. Analysis of molten pool images based on the change of robot welding gun inclination angle [J]. Automation and Information Engineering, 2020, 41(04): 6-12]. Wang Huan et al. disclosed the application of piecewise linear grayscale transformation in molten pool edge extraction. The grayscale value of the edge was determined by analyzing the grayscale histogram of the molten pool image, and then the complete molten pool edge was obtained by using the Canny operator to extract the edge [Wang Huan, Gao Xiangdong, Zhang Rui. Application of piecewise linear grayscale transformation in molten pool edge extraction [J]. Welding Technology, 2012, 41(4): 11-13]. Zhao Zhiyu et al. disclosed the image processing and feature extraction of laser welding molten pool based on machine vision. The obtained molten pool image was processed using median filtering, grayscale processing, Canny operator edge extraction, image enhancement and other steps to obtain welding image features [Zhao Zhiyu. Image processing and feature extraction of laser welding molten pool based on machine vision [J]. Applied Laser, 2021, 41(01): 195-200]. These are all welding monitoring methods based on image processing, which require manual adjustment of parameters. Incorrect settings may lead to unsatisfactory results. In addition, image processing methods are sensitive to noise in the image, and the presence of noise may affect the processing effect. In addition, image processing methods can only extract feature information of the image, but cannot give these features semantic information.

[0004] In recent years, deep learning technology has been widely used in the industrial field. Huang Junfen et al. disclosed the prediction of GMAW penetration state based on visual sensing. The molten pool image was trained by BP neural network to establish a molten pool state prediction model, and a binocular visual sensing welding molten pool monitoring system was built, which can monitor the molten pool morphology in real time and effectively predict the welding state [Huang Junfen, Xue Long, Huang Jiqiang, et al. Prediction of GMAW penetration state based on visual sensing [J]. Journal of Mechanical Engineering, 2019, 55(17): 41-47]. Fang Q et al. disclosed the on-site capture of molten pool features in the selective laser melting process based on U-Net convolutional neural network. By training the U-Net neural network, high-precision segmentation and positioning of the molten pool were achieved, and the Canny operator was used to extract the molten pool contour to optimize the segmentation quality [Fang Q, Tan Z, Li H, et al. In-situ capture of meltpool signature in selective laser melting usingU-Net-based convolutionalneural network[J]. Journal of Manufacturing Processes, 2021, 68: 347-355]. Ma Xiaofeng et al. disclosed a deep learning detection method for all-position welding molten pools, and proposed a deep learning detection method for molten pools that combines GAN and improved PSPNet to achieve dynamic tracking and accurate detection of the molten pool [Ma Xiaofeng, Xia Pan, Liu Haisheng, et al. Deep learning detection method for all-position welding molten pools [J]. Journal of Mechanical Engineering, 2023, 59(12): 272-283]. Based on the current progress in the research on molten pool image extraction, it can be seen that researchers have conducted in-depth exploration of molten pool image extraction using semantic segmentation methods based on deep learning. However, semantic segmentation requires precise annotation of each pixel, which is a very time-consuming and expensive process. Especially when a large-scale data set is required, the annotation cost is greatly increased. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a welding pool visual monitoring method based on multimodal semi-supervised semantic segmentation, which solves the time-consuming and costly problems caused by data labeling, and designs a novel multi-scale feature information enhancement processing method to improve the accuracy of the semantic segmentation network.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions.

[0007] The welding pool visual monitoring method based on multimodal semi-supervised semantic segmentation provided by the present invention comprises the following steps:

[0008] 1) Build a semantic segmentation network;

[0009] 2) Train the semantic segmentation network based on the semi-supervised semantic segmentation network training strategy. Step 2) includes the following sub-steps:

[0010] 21) Constructing a training set; the training set includes a labeled data set and a pseudo-labeled data set; the samples in the labeled data set are composed of an original image and a corresponding labeled image; the samples in the pseudo-labeled data set are composed of an original image and a corresponding first pseudo-labeled image;

[0011] 22) Constructing a student model and a teacher model; the student model and the teacher model are both constructed semantic segmentation networks;

[0012] 23) Train the teacher model using the labeled dataset;

[0013] 24) Input the original image in the pseudo-labeled dataset into the teacher model to obtain the corresponding second pseudo-labeled image;

[0014] 25) Randomly sample a batch of student models from the labeled data set and the pseudo-labeled data set, and calculate the total loss; then optimize the student model weights based on the total loss; the total loss calculation formula is L as follows:

[0015] ;

[0016] In the formula, L s represents the supervised loss calculated from the prediction results of the student model applied to the sample images in the labeled dataset and the label images, L u1 , L u2 They represent the unsupervised losses calculated from the prediction results of the student model applied to the sample images in the pseudo-labeled dataset and the first pseudo-labeled image and the second pseudo-labeled image respectively; , They represent the unsupervised loss L u1 , L u2 The weight of

[0017] 26) Update the teacher model weight according to the student model weight and return to step 24);

[0018] Repeat 24)-26) until the student model converges, and use the trained student model as the final semantic segmentation network;

[0019] 3) Input the collected original welding pool image into the trained semantic segmentation network to obtain the segmented welding pool image.

[0020] In the above step 1), the semantic segmentation network is a DDRNet (Deep Dual-Resolution Networks) network based on multi-shape information enhancement, which includes an input module, a first branch, a second branch and an output module; the output of the input module is respectively input into the first branch and the second branch, and the output of the first branch and the second branch are superimposed and input into the output module; the input module is used to perform preliminary feature extraction on the image input into the semantic segmentation network, and the input module includes one or more residual blocks; the first branch is used to obtain a high-resolution feature map based on the preliminary feature extraction result, the first branch includes one or more residual blocks and one or more residual bottleneck blocks, and each residual block or residual bottleneck block is provided with a multi-shape information enhancement block; the second branch is used to obtain a low-resolution feature map based on the preliminary feature extraction result, the second branch includes one or more residual blocks, one or more residual bottleneck blocks and a spatial pyramid pool SPP; the output module is used to fuse and output the output results of the first branch and the second branch.

[0021] In a specific implementation, the first branch includes a third residual block, a first multi-shape information enhancement block, a fourth residual block, a second multi-shape information enhancement block, a first residual bottleneck block, and a third multi-shape information enhancement block arranged in sequence; the second branch includes a fifth residual block, a sixth residual block, a second residual bottleneck block, and a spatial pyramid pool SPP arranged in sequence; the output module includes a segmentation head and an upsampling unit; the first multi-shape information enhancement block, the second multi-shape information enhancement block, and the third multi-shape information enhancement block have the same structure, and all include an input layer, a parallel processing layer, a splicing layer, and an output layer arranged in sequence; the parallel processing layer includes a horizontal strip convolution layer, a vertical strip convolution layer, an expansion convolution layer, and an n*n convolution layer arranged in parallel, and the convolution kernels of all the convolution layers arranged in parallel have the same size; the input layer and the output layer are both 1*1 convolution layers; the splicing layer is used to splice the outputs of all the convolution layers arranged in parallel together.

[0022] Furthermore, the above-mentioned input module also includes a first convolution block and a second convolution block arranged before the first residual block.

[0023] Furthermore, the output of the third residual block is also input into the sixth residual block, the output of the fifth residual block is also input into the fourth residual block, the output of the fourth residual block is also input into the second residual bottleneck block, and the output of the sixth residual block is also input into the first residual bottleneck block.

[0024] The goal of the above step 2) semi-supervised semantic segmentation is to train a semantic segmentation model by utilizing a large amount of unlabeled data and a small amount of labeled data. Assume that the entire training set contains N l Original tagged images and Nu Unlabeled original image ,in is the labeled original image, For K The label image of the class, is an unlabeled original image. Following the student-teacher framework, two networks are trained, namely the student model and teacher model , they share the same architecture (both the student model and the teacher model used here are DDRNet), but update their parameters separately. The specific training process is shown in steps 21)-26).

[0025] In the above step 21), the training set consists of labeled image data and unlabeled image data, so that only a small amount of labeled melt pool label data is needed to effectively train the semantic segmentation network.

[0026] The original images in the above labeled dataset and pseudo-labeled dataset are optical images, specifically visible light images. In the pseudo-labeled dataset, the first pseudo-labeled image is obtained based on infrared thermal imaging. First, the edge detection of the molten pool in the infrared thermal imaging is performed, and then the pixels inside and outside the molten pool are binarized to obtain the first pseudo-labeled image; the above edge detection can use the Canny edge detection operator.

[0027] In step 23), a teacher model is first pre-trained using a labeled dataset. The training method can be found in H. Pan, Y. Hong, W. Sun and Y. Jia, "Deep Dual-Resolution Networks for Real-Time and Accurate Semantic Segmentation of Traffic Scenes," in IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 3, pp. 3448-3460, March 2023, doi: 10.1109 / TITS.2022.3228042.

[0028] In the above step 24), the purpose is to obtain the second pseudo-label image corresponding to the original image in the pseudo-labeled dataset to facilitate the training of the student model.

[0029] no x j Each pixel has a corresponding pseudo label. For a given image xj ,use Indicates k Line l A threshold-based evaluation criterion inspired by FixMatch is used to establish a score for each pixel, denoted as S , which is the class-level distribution based on the teacher model prediction. If the score of a pixel is S Exceeding the threshold , then assign a pseudo label to the pixel:

[0030] ;

[0031] in, is the pixel probability of class c, and the score is set as .

[0032] In the above step 25), respectively from D l and D u Sampling a batch of B l and B u Image, the total loss of optimization objective can be written as:

[0033] .

[0034] L s , L u1 and L u2 Both are cross-entropy (CE) losses, specifically:

[0035] ;

[0036] ;

[0037] ;

[0038] in, represents the cross entropy function; , Respectively indicate from and Sample a batch of images from , Respectively , The number of images, , denote the labeled original image set and the unlabeled original image set respectively, and Respectively represent the student model to label the original image and the unlabeled original image predictions; Indicates i Original tagged images Label image of Indicates j Unlabeled original image The first pseudo-label image of Indicates j Unlabeled original image The second pseudo-labeled image.

[0039] According to the total loss, the student model weights are optimized through the gradient optimization algorithm.

[0040] In the above step 26), the weight of the teacher model is updated to the exponential moving average of the weight of the student model; specifically:

[0041] ;

[0042] In the formula, represents the teacher model weight of the nth cycle; represents the teacher model weight of the n-1th cycle; represents the student model weight of the nth cycle; Represents the weighting factor.

[0043] The weight of the teacher model in the nth cycle is obtained by weighting the weight of the teacher model in the n-1th cycle and the weight of the student model in the nth cycle, where α is 0.9.

[0044] The total loss or number of iterations of the student model can be used to determine whether the student model has converged. When the total loss is less than the set threshold or tends to be stable, the model can be determined to have converged; or when the number of iterations reaches the set maximum number of iterations, the student model is determined to have converged and the training is finished. The trained student model is used as the final semantic segmentation network.

[0045] Compared with the prior art, the welding pool visual monitoring method based on multimodal semi-supervised semantic segmentation provided by the present invention has the following beneficial effects:

[0046] 1) In the process of semantic segmentation network training, the present invention introduces infrared thermal imaging as a pseudo-label image, thereby realizing effective training of the semantic segmentation network based on a multimodal semi-supervised learning strategy combining supervised learning of a small amount of labeled data and unsupervised learning of pseudo-label data of different modalities, thereby improving the training efficiency of the semantic segmentation network and ensuring the accuracy of welding pool detection;

[0047] 2) The semantic segmentation network provided by the present invention, based on the traditional DDRNet network, introduces a multi-shape information enhancement block, which feeds feature maps from different convolution paths to achieve accurate extraction of molten pool image features, thereby helping to ensure the accuracy of electric welding molten pool detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 are molten pool images; where (a) is the visible light image collected during the two welding processes, (b) is the corresponding infrared thermal image in (a), and (c) is the label image of the molten pool corresponding to (a);

[0049] Figure 2 It is a flow chart of the welding pool visual monitoring method based on multimodal semi-supervised semantic segmentation;

[0050] Figure 3 A schematic diagram of a semantic segmentation network architecture constructed for an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of the multi-shape information enhancement block structure;

[0052] Figure 5 A schematic diagram of the principle of training a semantic segmentation network based on a semi-supervised semantic segmentation network training strategy;

[0053] Figure 6 Schematic diagram of obtaining the first pseudo-label image; wherein (a) is an infrared thermal image, (b) is a Canny edge detection result, and (c) is a binarization result (i.e., the first pseudo-label image);

[0054] Figure 7 This is the prediction result of the trained semantic segmentation network; (a), (d), (g) are original images; (b), (e), (h) are label images; (c), (f), (i) are prediction values. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions of various embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0056] Example

[0057] The welding pool visual monitoring method based on multimodal semi-supervised semantic segmentation provided by the present invention can be used in a welding system to realize the monitoring and identification of the welding pool in the collected image within the monitoring range of the welding pool.

[0058] In this embodiment, the welding system includes a German KUKA KR60 welding machine, in conjunction with a Japanese Yaskawa SK6 arc welding robot system, a welding current of 260A, a welding voltage of 30V, a welding speed of 30cm / min, and a shielding gas composition of 95%Ar+5%Co2. The base material is 10mm thick Q235 low-carbon steel, the welding wire is H08Mn2SiA, 1.2mm in diameter, and a V-shaped groove weld (60°, 2mm blunt edge). The molten pool visual sensing acquisition system includes a thermal imager with a temperature measurement range of -20~1000℃, equipped with visible light and infrared lenses, which can realize image fusion mode to view temperature data and save real-time images, such as Figure 1 As shown in (a) and (b).

[0059] In order to achieve Figure 1 (a) Identification and image segmentation of the molten pool in the visible light image. This embodiment provides a welding molten pool visual monitoring method based on multimodal semi-supervised semantic segmentation, such as Figure 2 As shown, it includes the following steps:

[0060] 1) Build a semantic segmentation network;

[0061] 2) Train the semantic segmentation network based on the semi-supervised semantic segmentation network training strategy;

[0062] 3) Input the collected original welding pool image into the trained semantic segmentation network to obtain the segmented welding pool image.

[0063] In the above step 1), the constructed semantic segmentation network is a DDRNet (Deep Dual-Resolution Networks) network based on multi-shape information enhancement. For the traditional DDRNet network, see H. Pan, Y. Hong, W. Sun and Y. Jia, "Deep Dual-Resolution Networks for Real-Time and Accurate Semantic Segmentation of Traffic Scenes," in IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 3, pp. 3448-3460, March 2023, doi:10.1109 / TITS.2022.3228042. The network consists of two branches, aiming to simultaneously extract high-resolution global feature information and high-level semantic details. This design enables it to have both powerful performance and excellent real-time performance. In addition, the DDRNet network has the characteristics of a limited number of parameters and low computational complexity. This embodiment involves a multi-shape information enhancement block (Multi-shape Information Enhancement Block, MIEBlock). The multi-shape information enhancement block is incorporated into DDRNet to form an improved DDRNet network.

[0064] The semantic segmentation network constructed in this embodiment is as follows: Figure 3 As shown, it includes an input module, a first branch, a second branch and an output module; the output of the input module is respectively input into the first branch and the second branch, and the output of the first branch and the second branch are superimposed and input into the output module; the input module is used to perform preliminary feature extraction on the image input into the semantic segmentation network; the first branch is used to obtain a high-resolution feature map based on the preliminary feature extraction result; the second branch is used to obtain a low-resolution feature map based on the preliminary feature extraction result; the output module is used to fuse the output results of the first branch and the second branch.

[0065] In this embodiment, the input module includes two first convolution blocks, a second convolution block, a first residual block, and a second residual block, which are arranged in sequence. The specific parameters are shown in Table 1.

[0066] The first branch includes a third residual block, a first multi-shape information enhancement block, a fourth residual block, a second multi-shape information enhancement block, a first residual bottleneck block, and a third multi-shape information enhancement block, which are arranged in sequence. The first branch is a high-resolution branch, in which the feature map contains more information and is more conducive to extracting the contour information of the target; therefore, in this embodiment, a multi-shape information enhancement block is inserted into the high-resolution branch. The first multi-shape information enhancement block, the second multi-shape information enhancement block, and the third multi-shape information enhancement block have the same structure, such as Figure 4 As shown, they all include an input layer, a parallel processing layer, a splicing layer and an output layer arranged in sequence; the parallel processing layer includes a horizontal strip convolution layer, a vertical strip convolution layer, an expanded convolution layer and an n*n convolution layer (n=3) arranged in parallel; the convolution kernel sizes of all the convolution layers arranged in parallel are the same, which are 9 in this embodiment, and the specific parameters are shown in Table 2. The input layer and the output layer are both 1*1 convolution layers. The splicing layer is used to splice the outputs of all the convolution layers arranged in parallel together. In this embodiment, the multi-shape information enhancement block feeds feature maps through four parallel paths after 1×1 convolution; then, the output feature maps of these paths are connected, and another 1×1 convolution is applied to generate the output of the multi-shape information enhancement module; so that the size of the feature map remains unchanged throughout the process.

[0067] The second branch includes the fifth residual block, the sixth residual block, the second residual bottleneck block, and the spatial pyramid pool SPP (using the original deep aggregation pyramid pool module of DDRNet); the output module includes a segmentation head and an upsampling unit. The specific parameters of the second branch are shown in Table 1.

[0068] Moreover, the output of the third residual block in the first branch is also input into the sixth residual block, and the output of the fourth residual block is also input into the second residual bottleneck block; the output of the fifth residual block in the second branch is also input into the fourth residual block, and the output of the sixth residual block is also input into the first residual bottleneck block.

[0069] Table 1 Detailed information of semantic segmentation network parameters

[0070]

[0071] Note: Assume that the input image size is 896 × 896 × 3; N in the table represents the number of types of objects to be detected, N=2.

[0072] Table 2 Parameter details of the multi-shape information enhancement block.

[0073]

[0074] In step 2 above, the goal of semi-supervised semantic segmentation is to train a semantic segmentation model by utilizing a large amount of unlabeled data and a small amount of labeled data. Assume that the entire training set contains Nl Original tagged images and N u Unlabeled original image ,in is the labeled original image, For K The label image of the class, is an unlabeled original image. Following the student-teacher framework, two networks are trained, namely the student model and teacher model , they share the same architecture (both the student model and the teacher model used here are DDRNet), but update their parameters separately.

[0075] Step 2 above), such as Figure 5 As shown, it includes the following sub-steps:

[0076] 21) Constructing a training set; the training set includes a labeled data set and a pseudo-labeled data set; the samples in the labeled data set are composed of an original image and a corresponding labeled image; the samples in the pseudo-labeled data set are composed of an original image and a corresponding first pseudo-labeled image.

[0077] Typical melt pool images collected are as follows: Figure 1 As shown, the molten pool area is the brightest white area, and the entire molten pool image is approximately "circular". In this embodiment, a total of 1,000 original molten pool images were collected, and the cropped images were 512×512 in size, all in jpg format. The molten pool images were then rotated, cropped, flipped, and translated using Pytorch's image enhancement library Augmentor to expand the data set to 5,000 images. The training set and the test set are divided into a ratio of 8:2.

[0078] In the training set, 2000 original images are labeled. These original images and their corresponding labeled images constitute the labeled data set; the labeled labeled images are as follows: Figure 1 (c). Figure 6 As shown in FIG. 1 , for other unlabeled original images, the corresponding infrared thermal images are subjected to the Canny edge detection operator to detect the edge of the molten pool, and then the pixels inside and outside the detected edge are binarized to obtain the first pseudo-label image. All unlabeled original images and the corresponding first pseudo-label images constitute the pseudo-label dataset.

[0079] 22) Constructing a student model and a teacher model; the student model and the teacher model are both constructed semantic segmentation networks.

[0080] The constructed semantic segmentation network is as described above.

[0081] 23) Train the teacher model using the labeled dataset.

[0082] First, use the labeled data set to pre-train a teacher model. The training method can refer to the training process given in the previous reference.

[0083] 24) Input the original image in the pseudo-labeled dataset into the teacher model to obtain the corresponding second pseudo-labeled image.

[0084] The purpose here is to obtain the second pseudo-label image corresponding to the original image in the pseudo-labeled dataset to facilitate the training of the student model.

[0085] no x j Each pixel has a corresponding pseudo label. For a given image x j ,use Indicates k Line l A threshold-based evaluation criterion inspired by FixMatch is used to establish a score for each pixel, denoted as S , which is the class-level distribution based on the teacher model prediction. If the score of a pixel is S Exceeding the threshold , then assign a pseudo label to the pixel:

[0086]

[0087] in, is the pixel probability of class c, and the score is set as .

[0088] In this example, there is only one category, the melt pool. Therefore, if the score of a pixel is S Exceeding the threshold , then the pixel probability predicted by the teacher model is used as the second pseudo label image.

[0089] 25) Randomly sample a batch of student models from the labeled data set and the pseudo-labeled data set, and calculate the total loss; then optimize the student model weights based on the total loss; the total loss calculation formula is L as follows:

[0090] ;

[0091] In the formula, L s represents the supervised loss calculated from the prediction results of the student model applied to the sample images in the labeled dataset and the label images, L u1 ,L u2 They represent the unsupervised losses calculated from the prediction results of the student model applied to the sample images in the pseudo-labeled dataset and the first pseudo-labeled image and the second pseudo-labeled image respectively; , They represent the unsupervised loss L u1 , L u2 The weight of .

[0092] In this embodiment, the number of original images randomly sampled from the labeled dataset and the pseudo-labeled dataset is the same.

[0093] L s , L u1 and L u2 Both are cross-entropy (CE) losses, specifically:

[0094] ;

[0095] ;

[0096] ;

[0097] in, represents the cross entropy function; , Respectively indicate from and Sample a batch of images from , Respectively , The number of images, , denote the labeled original image set and the unlabeled original image set respectively, and Respectively represent the student model to label the original image and the unlabeled original image predictions; Indicates i Original tagged images Label image of Indicates j Unlabeled original image The first pseudo-label image of Indicates j Unlabeled original image The second pseudo-labeled image.

[0098] According to the total loss, the student model weights are optimized through the gradient optimization algorithm.

[0099] Update the weights of the teacher model to the exponential moving average of the student model weights; specifically:

[0100] ;

[0101] In the formula, represents the teacher model weight of the nth cycle; represents the teacher model weight of the n-1th cycle; represents the student model weight of the nth cycle; Represents the weighting factor.

[0102] The weight of the teacher model in the nth cycle is obtained by weighting the weight of the teacher model in the n-1th cycle and the weight of the student model in the nth cycle, where α is 0.9.

[0103] 26) Update the teacher model weights based on the student model weights and return to step 24).

[0104] Since the teacher model weights are updated at this time, it is necessary to return to step 24) to regenerate a more realistic second pseudo-label image to more effectively train the student model.

[0105] Repeat 24)-26) until the student model converges, and use the trained student model as the final semantic segmentation network;

[0106] In this embodiment, the loss function is used to determine whether the student model has converged. When the loss function tends to be stable, it can be determined that the model has converged. The trained student model is used as the final semantic segmentation network.

[0107] In this embodiment, the original image of the melt pool in the test set is input into the trained final semantic segmentation network, and the final semantic segmentation network is tested. The test results are shown in Figure 7 As shown. Figure 7 It can be seen that the predicted values ​​of the proposed method are relatively close to the true labels, indicating that the method has high accuracy and effectiveness in data analysis and model building.

[0108] Those skilled in the art will appreciate that the embodiments herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A welding pool visual monitoring method based on multimodal semi-supervised semantic segmentation, It is characterized in that The following steps are involved: 1) Constructing a semantic segmentation network; the semantic segmentation network is a DDRNet network based on multi-shape information enhancement, including an input module, a first branch, a second branch and an output module; the output of the input module is respectively input into the first branch and the second branch, and the output of the first branch and the second branch are superimposed and input into the output module; the input module is used to perform preliminary feature extraction on the image input into the semantic segmentation network, and the input module includes one or more residual blocks; the first branch is used to obtain a high-resolution feature map based on the preliminary feature extraction result, the first branch includes one or more residual blocks and one or more residual bottleneck blocks, and each residual block or residual bottleneck block is provided with a multi-shape information enhancement block; the second branch is used to obtain a low-resolution feature map based on the preliminary feature extraction result, and the second branch includes one or more residual blocks, one or more residual bottleneck blocks and a spatial pyramid pool SPP; the output module is used to fuse the output results of the first branch and the second branch; 2) Train the semantic segmentation network based on the semi-supervised semantic segmentation network training strategy. Step 2) includes the following sub-steps: 21) Constructing a training set; the training set includes a labeled data set and a pseudo-labeled data set; the samples in the labeled data set are composed of an original image and a corresponding labeled image; the samples in the pseudo-labeled data set are composed of an original image and a corresponding first pseudo-labeled image; 22) Constructing a student model and a teacher model; the student model and the teacher model are both constructed semantic segmentation networks; 23) Train the teacher model using the labeled dataset; 24) Input the original image in the pseudo-labeled dataset into the teacher model to obtain the corresponding second pseudo-labeled image; 25) Randomly sample a batch of student models from the labeled data set and the pseudo-labeled data set, and calculate the total loss; then optimize the student model parameters based on the total loss; the total loss calculation formula is L as follows: ； In the formula, L s represents the supervised loss calculated from the prediction results of the student model applied to the sample images in the labeled dataset and the label images, L u1 , L u2 They represent the unsupervised losses calculated from the prediction results of the student model applied to the sample images in the pseudo-labeled dataset and the first pseudo-labeled image and the second pseudo-labeled image respectively; , They represent the unsupervised loss L u1 , L u2 The weight of 26) Update the teacher model weight according to the student model weight and return to step 24); Repeat 24)-26) until the student model converges, and use the trained student model as the final semantic segmentation network; 3) Input the collected original welding pool image into the trained semantic segmentation network to obtain the segmented welding pool image.

2. According to claim 1, the welding pool visual monitoring method based on multimodal semi-supervised semantic segmentation, It is characterized in that The input module includes two first residual blocks and a second residual block arranged in sequence; the first branch includes a third residual block, a first multi-shape information enhancement block, a fourth residual block, a second multi-shape information enhancement block, a first residual bottleneck block, and a third multi-shape information enhancement block arranged in sequence; the second branch includes a fifth residual block, a sixth residual block, a second residual bottleneck block, and a spatial pyramid pool SPP arranged in sequence; the output module includes a segmentation head and an upsampling unit; the first multi-shape information enhancement block, the second multi-shape information enhancement block, and the third multi-shape information enhancement block have the same structure, and all include an input layer, a parallel processing layer, a splicing layer, and an output layer arranged in sequence; the parallel processing layer includes a horizontal strip convolution layer, a vertical strip convolution layer, an expansion convolution layer, and an n*n convolution layer arranged in parallel, and the convolution kernels of all the convolution layers arranged in parallel have the same size; the input layer and the output layer are both 1*1 convolution layers; the splicing layer is used to splice the outputs of all the convolution layers arranged in parallel together.

3. The welding pool visual monitoring method based on multimodal semi-supervised semantic segmentation according to claim 2, It is characterized in that The input module also includes a first convolution block and a second convolution block arranged before the first residual block.

4. The welding pool visual monitoring method based on multimodal semi-supervised semantic segmentation according to claim 2, It is characterized in that The output of the third residual block is also input into the sixth residual block, the output of the fifth residual block is also input into the fourth residual block, the output of the fourth residual block is also input into the second residual bottleneck block, and the output of the sixth residual block is also input into the first residual bottleneck block.

5. The welding pool visual monitoring method based on multimodal semi-supervised semantic segmentation according to any one of claims 1 to 4, It is characterized in that The original images in the labeled dataset and pseudo-labeled dataset are visible light images.

6. The welding pool visual monitoring method based on multimodal semi-supervised semantic segmentation according to claim 5, It is characterized in that In the pseudo-labeled dataset, the first pseudo-labeled image is obtained based on infrared thermal imaging. First, the edge of the molten pool in the infrared thermal imaging is detected, and then the pixels inside and outside the molten pool are binarized to obtain the first pseudo-labeled image.

7. The welding pool visual monitoring method based on multimodal semi-supervised semantic segmentation according to claim 5, It is characterized in that In step 25), L s , L u1 and L u2 They are all cross entropy losses, specifically: ； ； ； in, represents the cross entropy function; , Respectively indicate from and Sample a batch of images from , Respectively , The number of images, , denote the labeled original image set and the unlabeled original image set respectively, and Respectively represent the student model to mark the original image and the unlabeled original image predictions; Indicates i Original tagged images Label image of Indicates j Unlabeled original image The first pseudo-label image of Indicates j Unlabeled original image The second pseudo-labeled image.

8. The welding pool visual monitoring method based on multimodal semi-supervised semantic segmentation according to claim 5, It is characterized in that In step 26), the weights of the teacher model are updated to the exponential moving average of the weights of the student model.

9. The welding pool visual monitoring method based on multimodal semi-supervised semantic segmentation according to claim 8, It is characterized in that The weight update formula of the teacher model is: ； In the formula, represents the teacher model weight of the nth cycle; represents the teacher model weight of the n-1th cycle; represents the student model weight of the nth cycle; Represents the weighting factor.

Images