Welding seam defect detection method, evaluation method, equipment, storage medium and product

Abstract

The invention discloses a weld defect detection method, an evaluation method, equipment, a storage medium and a product. The detection method comprises the following steps: constructing a deep convolutional generative adversarial network and a weld defect detection model; wherein the deep convolutional generative adversarial network comprises a generator and a first discriminator, and the weld defect detection model adopts a U-Net network and comprises an encoder, a decoder, a newly added splicing layer and a second discriminator having the same architecture as the first discriminator; training the deep convolutional generative adversarial network by using the first sample data set to obtain a weight parameter of a first discriminator; transferring the weight parameter of the first discriminator to a second discriminator, and training the U-Net network by using the second sample data set to obtain a target defect detection model; and detecting the actual weld defect image by using the target defect detection model to obtain a weld segmentation result and a defect segmentation result. According to the method, the dependence on the labeled data is reduced, and meanwhile, the welding seam defect detection precision is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a weld defect detection method, evaluation method, equipment, storage medium and product based on semi-supervised transfer learning and feature fusion. Background Art

[0002] In industrial production, the integrity of welds directly affects the structural safety of products. If defects in welds (such as pores, cracks, etc.) are not detected and handled in time, they may pose serious safety hazards. Current weld defect detection technology mainly relies on manual detection and automated detection methods based on image processing. However, manual detection is inefficient and easily affected by the subjective judgment of inspectors, resulting in inconsistent detection results. In addition, traditional image processing technology has performance bottlenecks when facing complex backgrounds and subtle defects in weld images, and it is difficult to adapt to the imbalanced data in actual industrial environments.

[0003] In recent years, with the rapid development of deep learning technology, automated weld defect detection methods based on convolutional neural networks (CNNs) have gradually become mainstream. Deep learning models such as CNNs can automatically extract features from images, especially in the semantic segmentation task of weld images, such as the application of the U-Net network architecture, which has achieved certain results. However, deep learning models generally rely on a large amount of labeled data for training, and in industrial weld inspection scenarios, it is often very difficult and time-consuming to obtain labeled data. Therefore, in the case of insufficient labeled data, existing deep learning models are prone to overfitting, which affects the generalization ability of the model, especially when dealing with different types of weld defects.

[0004] In addition, the defects in welds are usually small in size and unevenly distributed, with a large difference in pixel ratio compared to the weld itself. This data imbalance problem causes the deep learning model to have weak detection capabilities in small defect areas, which is prone to missed detection or false detection. In addition, there are various types of weld defects, including pores, cracks, and lack of fusion. Traditional deep learning models find it difficult to maintain high detection accuracy in weld images with complex backgrounds and different defect morphologies. Although data augmentation technology can alleviate the problem of insufficient data to a certain extent, choosing a suitable data augmentation method is still a big challenge.

[0005] In summary, the existing weld defect detection solutions, especially in complex industrial environments, still have the following significant shortcomings:

[0006] (1) Strong dependence on labeled data: Existing deep learning models require a large amount of labeled data for training. When labeled data is scarce, the model’s generalization ability is insufficient and overfitting is prone to occur. In industrial scenarios, labeling large-scale data is time-consuming and laborious, and the acquisition process is extremely difficult.

[0007] (2) Data imbalance problem: Weld defects are usually small and rare, and the pixel distribution is seriously unbalanced with the weld background, resulting in low detection accuracy of the deep learning model for the defect area, which is prone to missed detection and false detection.

[0008] (3) Diversity and complexity of defect detection: There are many types of weld defects (including pores, cracks, etc.) with different shapes. Especially in complex industrial backgrounds, traditional detection methods are difficult to adapt to these diverse defect shapes and lack robustness.

[0009] Although some studies have attempted to improve the performance of weld image semantic segmentation by improving deep learning architectures (such as U-Net networks), these methods still perform poorly when dealing with data imbalance, small samples, and complex backgrounds. In particular, the accuracy and robustness of existing methods are still not ideal in terms of small defect detection. Therefore, how to improve the accuracy and efficiency of weld defect detection under the conditions of limited labeled data, uneven defect distribution, and complex background has become an urgent problem to be solved. Summary of the invention

[0010] The purpose of the present invention is to provide a weld defect detection method, evaluation method, equipment, storage medium and product to solve at least one of the problems that traditional weld defect detection technology based on deep learning models has a strong dependence on labeled data, the data imbalance of weld defects leads to low detection accuracy of traditional methods, and the diversity and complexity of weld defect detection leads to poor robustness of traditional methods.

[0011] The present invention solves the above technical problems through the following technical solutions: a weld defect detection method, comprising:

[0012] Acquire a first sample data set and a second sample data set; wherein each sample in the first sample data set includes a weld image, and each sample in the second sample data set includes a weld image and its weld area and defect area;

[0013] Constructing a deep convolutional generative adversarial network and a weld defect detection model; wherein the deep convolutional generative adversarial network includes a generator and a first discriminator; the weld defect detection model adopts a U-Net network, the U-Net network includes an encoder, a decoder, a newly added splicing layer, and a second discriminator having the same architecture as the first discriminator, the second discriminator is connected to the bottleneck layer in the encoder, and the newly added splicing layer is connected to the bottleneck layer, the second discriminator and the decoder in the encoder;

[0014] Using the first sample data set to train a deep convolutional generative adversarial network to obtain weight parameters of a first discriminator;

[0015] Migrating the weight parameters of the first discriminator to the second discriminator, and then training the U-Net network using the second sample data set to obtain a target defect detection model;

[0016] An actual weld defect image is acquired, and the target defect detection model is used to detect the actual weld defect image to obtain a weld segmentation result and a defect segmentation result.

[0017] Furthermore, the specific process of obtaining the first sample data set or the second sample data set includes:

[0018] A series of weld images of the structure are collected by an industrial camera; wherein the series of weld images include normal weld images and weld defect images of different defect types;

[0019] Performing a first image enhancement process on each of the weld images, and constructing a first sample data set based on the images after the first image enhancement process;

[0020] Each weld image is labeled and subjected to second image enhancement processing in turn, and a second sample data set is constructed based on the image after the second image enhancement processing.

[0021] Further, the first image enhancement processing includes affine transformation, random cropping, unpaired image defogging, elastic transformation, flipping, grid distortion, perspective transformation, adding Gaussian noise, adding ISO noise, image compression, random brightness contrast, and grayscale;

[0022] The second image enhancement process includes adjusting illumination, adding noise, adding random points, moving images and bounding boxes, and flipping.

[0023] Further, the U-Net network is trained using the second sample data set, specifically including:

[0024] Using an encoder to extract features from each weld image in the second sample data set to obtain a first feature value;

[0025] Using the bottleneck layer of the encoder to perform channel compression on the first feature quantity to obtain a second feature quantity;

[0026] Using the second discriminator to extract features from the second feature quantity to obtain a third feature quantity;

[0027] Using the newly added splicing layer to splice the first feature quantity and the third feature quantity to obtain a fourth feature quantity;

[0028] Using the decoder to extract features from the fourth feature quantity to obtain a region segmentation result;

[0029] A loss value is calculated according to the region segmentation result and the corresponding weld region and defect region, and a weight parameter of the U-Net network is adjusted according to the loss value.

[0030] Based on the same concept, the present invention provides a weld safety assessment method, comprising:

[0031] Using the weld defect detection method as described above to obtain weld segmentation results and defect segmentation results;

[0032] Determine the geometric parameters of the weld area according to the weld segmentation result; determine the number of defect areas and the geometric parameters of each defect area according to the defect segmentation result;

[0033] The weld safety evaluation is performed based on the geometric parameters of the weld area, the number of defect areas and the geometric parameters of each defect area.

[0034] Further, the geometric parameters of the weld area are determined according to the weld segmentation result, specifically including:

[0035] Determine the minimum circumscribed rectangle of the weld area according to the weld segmentation result, and determine the geometric parameters of the weld area according to the minimum circumscribed rectangle of the weld area;

[0036] Determining the geometric parameters of each defect area according to the defect segmentation result specifically includes:

[0037] The minimum circumscribed figure of each defect area is determined according to the defect segmentation result, and the geometric parameters of the corresponding defect area are determined according to the minimum circumscribed figure of each defect area.

[0038] Further, a weld safety assessment is performed based on the geometric parameters of the weld area, the number of defective areas and the geometric parameters of each defective area, including:

[0039] Determine a reference value according to the geometric parameters of the weld area, and divide the weld area into a plurality of sub-areas according to the reference value;

[0040] Conduct weld safety evaluation of the corresponding sub-area according to the number of defective areas in each sub-area and the geometric parameters of each defective area;

[0041] Among them, the weld safety evaluation of the corresponding sub-area is performed according to the number of defective areas in each sub-area and the geometric parameters of each defective area, including:

[0042] When the geometric parameters of each defective area in the sub-area are 0, the weld of the corresponding sub-area is safe;

[0043] When the number of defective areas in a sub-area is less than or equal to the number threshold, and the geometric parameter of each defective area is less than or equal to the geometric parameter threshold, the weld of the corresponding sub-area is safe;

[0044] When the number of defective areas in a sub-area is less than or equal to the number threshold, and the geometric parameter of each defective area is greater than the geometric parameter threshold, the weld in the corresponding sub-area is unsafe;

[0045] When the number of defective areas in a sub-area is greater than a quantity threshold, the weld in the corresponding sub-area is unsafe.

[0046] Based on the same concept, the present invention also provides an electronic device, including a memory, a processor, and a computer program / instructions stored in the memory, wherein the processor executes the computer program / instructions to implement the weld defect detection method or weld safety evaluation method as described above.

[0047] Based on the same concept, the present invention also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the weld defect detection method or weld safety evaluation method as described above is implemented.

[0048] Based on the same concept, the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the weld defect detection method or weld safety assessment method as described above.

[0049] Beneficial Effects

[0050] Compared with the prior art, the advantages of the present invention are:

[0051] A weld defect detection method provided by the present invention learns deep features from a large amount of unlabeled data through a deep convolutional generative adversarial network, and uses transfer learning to transfer the discriminator weight parameters obtained by the deep convolutional generative adversarial network to the discriminator of the U-Net network, thereby reducing the dependence on labeled data and improving the defect detection accuracy, and effectively alleviating the problem of insufficient and difficult to obtain labeled data; the present invention uses the newly added discriminator in the U-Net network to further extract features extracted by the encoder, and splices the further extracted features with the features extracted by the encoder through a newly added splicing layer, and then decodes them by the decoder, thereby enhancing the U-Net network's learning of complex features in weld images, which is more conducive to the detection of diverse defects, complex defects and / or subtle defects, and improves the weld defect detection accuracy and detection robustness.

[0052] A weld safety evaluation method provided by the present invention performs weld safety evaluation based on the geometric parameters of the weld area, the number of defect areas and the geometric parameters of each defect area, thereby solving the data imbalance problem caused by the small size and uneven distribution of weld defects and improving the accuracy of weld safety evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only one embodiment of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 is a flow chart of a weld defect detection method according to an embodiment of the present invention;

[0055] Figure 2 : is a weld image in an embodiment of the present invention; wherein (a) is an original weld image captured by an industrial camera, (b) is a cropped weld image, (c) is an annotated weld image, and (d) is a grayscale weld image;

[0056] Figure 3 is a diagram of the architecture of a deep convolutional generative adversarial network in an embodiment of the present invention;

[0057] Figure 4 is a multi-domain network training flow chart based on semi-supervised transfer learning and feature fusion in an embodiment of the present invention;

[0058] Figure 5 This is the original U-Net network architecture diagram in an embodiment of the present invention;

[0059] Figure 6 It is a flow chart comparing the target defect detection model in an embodiment of the present invention with the classic U-Net network;

[0060] Figure 7 : is a Dice score comparison curve of the three models of STLF-MDN, original U-Net and AUnet in the embodiment of the present invention;

[0061] Figure 8 : This is the IoU score comparison curve of the three models of STLF-MDN, original U-Net and AUnet in the embodiment of the present invention;

[0062] Fig. 9 This is a weld safety assessment process when there are porosity defects in an embodiment of the present invention;

[0063] Fig.10 1 is the STLF-MDN segmentation and evaluation result of different weld images in the embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following is a clear and complete description of the technical solutions in the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0066] Existing weld defect detection technologies, especially deep learning methods based on convolutional neural networks (CNNs), show significant deficiencies when dealing with high noise backgrounds and small defects. First, these methods are highly dependent on large-scale annotated data sets. In industrial weld detection, it is difficult to obtain annotated data, which leads to easy overfitting of the model during the training process and a significant decrease in detection accuracy. Secondly, they lack sufficient flexibility in dealing with irregular shapes and different sizes of weld defects, making it difficult to effectively capture multi-scale defect features, especially in the detection of small-sized defects. Finally, in the case of data imbalance, traditional deep learning models often ignore small-sized defects, resulting in a high missed detection rate. At the same time, these methods require cumbersome parameter adjustment and a long training cycle, which makes it difficult to meet the requirements of real-time and high efficiency in industrial applications. Therefore, the existing technology still has obvious deficiencies in accuracy, speed and adaptability, especially in demanding weld detection applications such as steel bridge decks, where its limitations are more prominent. Based on this, the present invention provides a weld defect detection method and evaluation method based on semi-supervised transfer learning and feature fusion.

[0067] Example 1

[0068] like Figure 1 As shown, a weld defect detection method provided by an embodiment of the present invention comprises the following steps:

[0069] Step 1: Obtain a first sample data set and a second sample data set.

[0070] In a specific embodiment of the present invention, the specific process of obtaining the first sample data set or the second sample data set includes:

[0071] Step 1.1: Collect a series of weld images of the structure by using an industrial camera; wherein the series of weld images include normal weld images and weld defect images of different defect types;

[0072] Step 1.2: Perform a first image enhancement process on each weld image, and construct a first sample data set based on the image after the first image enhancement process;

[0073] Step 1.3: Annotate and perform second image enhancement processing on each weld image in turn, and construct a second sample data set based on the images after the second image enhancement processing.

[0074] Each sample in the first sample data set includes a weld image, and each sample in the second sample data set includes a weld image and its weld area and defect area, that is, the first sample data set is an unlabeled data set, and the second sample data set is a labeled data set. In this embodiment, the first image enhancement processing includes affine transformation, random cropping, unpaired image defogging (such as 4D method), elastic transformation, flipping, grid distortion, perspective transformation, adding Gaussian noise, adding ISO noise, image compression, random brightness contrast, and grayscale; the second image enhancement processing includes illumination adjustment, adding noise, adding random points, moving images and bounding boxes, and flipping.

[0075] In order to solve the problem of strong dependence on labeled data sets, the present invention obtains two data sets, one is an unlabeled data set and the other is a labeled data set, and migrates the weight file generated by learning the features of the unlabeled data set to the weld defect detection model through transfer learning, which can improve the accuracy of weld defect detection while reducing the dependence on the labeled data set. In order to solve the problem of data scarcity, the present invention performs a first image enhancement process and a second image enhancement process on each weld image collected by an industrial camera, and expands the diversity of data while ensuring that the mask remains unchanged through regional and pixel-level image enhancement technology, effectively alleviating the problem of data scarcity.

[0076] For example, an industrial camera is used to obtain a series of weld seam images of a steel bridge deck. The image shooting distance is about 20 cm, and the resolution of the collected original images is 4000×1846 pixels. Figure 2 These images were then cropped to a size of 1024 × 1024 pixels (see Figure 2 (b) in the figure), and manually use different colors to mark different areas, such as Figure 2 As shown in (c) in the figure, the annotated color image is then converted into a grayscale label image and divided into three categories pixel by pixel: the pixel value 0 represents the background, 50 represents the weld area, and 255 represents the pore category, as shown in Figure 2 As shown in (d) in the figure, the types of the original weld images collected are shown in Table 1, and the weld defect types include pores.

[0077] Table 1 Weld image types

[0078]

[0079] The first sample data set and the second sample data set generated after the first image enhancement processing and the second image enhancement processing are shown in Table 2 and Table 3.

[0080] Table 2 The first sample data set

[0081]

[0082] Table 3 The second sample data set

[0083]

[0084] Step 2: Build a deep convolutional generative adversarial network and weld defect detection model.

[0085] like Figure 3 As shown in FIG, the deep convolutional generative adversarial network (DCGAN model) includes a generator and a first discriminator. The weld defect detection model adopts a U-Net network. The U-Net network includes an encoder, a decoder, a newly added splicing layer, and a second discriminator with the same architecture as the first discriminator. The second discriminator is connected to the bottleneck layer in the encoder, and the newly added splicing layer is connected to the bottleneck layer in the encoder, the second discriminator, and the decoder. Figure 4 As shown. The U-Net network of the present invention is the original U-Net network (such as Figure 5 The variant of the proposed method is shown in FIG. 1 , that is, based on the original U-Net network, a second discriminator and a concatenation layer are added to the bottleneck layer of the encoder, the second discriminator is used to further extract the features output by the bottleneck layer, and the newly added concatenation layer is used to concatenate (i.e., fuse) the features output by the bottleneck layer and the features output by the second discriminator, and then the decoder decodes the concatenated features. Figure 4 The multi-domain network based on semi-supervised transfer learning and feature fusion in the method refers to using a first sample data set to perform unsupervised training on a deep convolutional generative adversarial network to obtain weight parameters of a first discriminator; the weight parameters of the first discriminator are transferred to the second discriminator of a U-Net network through transfer learning, and finally the U-Net network is supervisedly trained using a second sample data set to obtain a network.

[0086] Step 3: Use the first sample data set to train the deep convolutional generative adversarial network to obtain the weight parameters of the first discriminator.

[0087] Through unsupervised training, the deep convolutional generative adversarial network extracts deep semantic features from the unlabeled dataset (i.e., the first sample dataset), including edge, texture, and shape features in the weld image. These features are reflected in the weight parameters of the first discriminator. The weight parameters of the first discriminator are then transferred to the second discriminator, which can improve the U-Net network's ability to capture image details at different scales, and further improve the semantic segmentation accuracy and the generalization ability of the U-Net network.

[0088] Step 4: Transfer the weight parameters of the first discriminator to the second discriminator, and then use the second sample data set to train the U-Net network to obtain the target defect detection model.

[0089] In a specific embodiment of the present invention, the U-Net network is trained using the second sample data set, specifically including:

[0090] Step 4.1: Using an encoder to extract features from each weld image in the second sample data set to obtain a first feature value;

[0091] Step 4.2: Use the bottleneck layer of the encoder to perform channel compression on the first feature quantity to obtain a second feature quantity;

[0092] Step 4.3: Use the second discriminator to extract features from the second feature quantity to obtain a third feature quantity;

[0093] Step 4.4: Use the newly added splicing layer to splice the first feature quantity and the third feature quantity to obtain a fourth feature quantity;

[0094] Step 4.5: Use the decoder to extract features from the fourth feature quantity to obtain a region segmentation result;

[0095] Step 4.6: Calculate the loss value based on the region segmentation result and the corresponding weld area and defect area, and adjust the weight parameters of the U-Net network according to the loss value.

[0096] The bottleneck layer of the encoder performs channel compression on the input 64×64×1024 high-dimensional feature map (i.e., the first feature quantity) through a 1×1 convolution operation, reducing the number of channels to 64×64×3 (i.e., the second feature quantity) to ensure that the second feature quantity output by the bottleneck layer is compatible with the input size of the second discriminator. This channel compression strategy not only reduces the computational complexity, but also ensures the transmission of key information by selectively retaining channels with high information content. The second feature quantity is input to the second discriminator, which uses its hierarchical convolution structure to perform layer-by-layer dimensionality reduction and high-order feature extraction on the second feature quantity, and finally generates a 4×4×512 high-level feature map (i.e., the third feature quantity). These high-level feature maps contain high-level semantic information after multiple convolutions, and have strong category distinction and fine structure recognition capabilities.

[0097] In order to effectively fuse these high-level feature maps (i.e., the third feature quantity) with the multi-scale features (i.e., the first feature quantity) output by the encoder, the present invention uses a bilinear interpolation method to spatially upsample the 4×4×512 high-level feature map output by the second discriminator, restoring it to a resolution of 64×64×512. Through the bilinear interpolation method, the high-order semantic information is expanded to a scale that matches the original feature map, ensuring the spatial consistency between feature maps. After obtaining the same spatial dimension, the feature map with a resolution of 64×64×512 is spliced ​​with the 64×64×1024 high-dimensional feature map originally output by the bottleneck layer in the channel dimension to form a 64×64×1536 spliced ​​feature map (i.e., the fourth feature quantity). Through splicing, the model can flexibly capture weld defects of different scales and shapes, significantly improving the detection accuracy and robustness, especially in the detection of small defects. In order to avoid the introduction of redundant information and enhance the expressiveness of features, a 1×1 convolution kernel is used to perform deep convolution operation on the spliced ​​feature map, compressing its number of channels to 1024 to match the input requirements of the decoder.

[0098] The target defect detection model (i.e., the multi-domain network STLF-MDN (Semi-Supervised Transfer Learning-Based Multi-Domain Network with FeatureFusion) based on semi-supervised transfer learning and feature fusion) can still achieve efficient weld defect detection even when the labeled data is limited. It performs well in dealing with small-size defects and complex backgrounds, avoiding the overfitting problem and improving the generalization ability of the model.

[0099] In order to verify the robustness and accuracy of the target defect detection model, the segmentation results of the target defect detection model are compared with the segmentation results of multiple classic U-Net networks, such as Figure 6 By comparing the key performance indicators of each model (such as IoU (intersection over union) and Dice score), the best segmentation model is determined to be the target defect detection model of the present invention.

[0100] The five comparison models and the target defect detection model of the present invention were trained after applying the "early stopping" callback to obtain the best Dice score and IoU score, and the results are shown in Table 4. For the target defect detection model of the present invention, before training, the weight parameters extracted from the first discriminator (DCGAN was trained on the first sample data set and the callback was applied, and the loss of the first discriminator was reduced to 0.05) were first transferred to the second discriminator of the U-Net network of the present invention; then, the U-Net network of the present invention was trained in the second sample data set. According to the test IoU score, three best segmentation models were finally selected for further analysis, namely the target defect detection model of the present invention STLF-MDN, the original U-Net, and AUnet (Attention Unet).

[0101] Table 4 Comparison of performance indicators of the target defect detection model of the present invention and the comparative model

[0102] Serial number Model IoU score Dice score Number of training rounds 1 STLF-MDN 93.3 0.917 174 2 Original Unet 89.4 0.881 161 3 AUnet 90.7 0.899 225 4 U-Net++ 83.8 0.882 158 5 ResUNet 84.1 0.884 162 6 R2U-Net 81.1 0.875 142

[0103] Due to the early stopping strategy, the number of training rounds for each model is different. In order to fairly compare the segmentation performance of each model, the indicator data when training to 160 rounds was selected, and the performance of the three models of STLF-MDN, original U-Net and AUnet was further analyzed. The results are as follows Figure 7 and Figure 8 Compared with the U-Net and AUnet models, STLF-MDN's Dice scores increased by 4.6% and 4.7%, and IoU scores increased by 9.1% and 11.2%, respectively; in addition, the original U-Net is fastest to train when it reaches an IoU score of 90. From the Dice curve ( Figure 7 ) It can be seen that the STLF-MDN model captures effective features faster than other models during the training process, thereby achieving a higher performance level at an earlier stage, and the curve also tends to be stable earlier. This shows that the STLF-MDN model of the present invention greatly improves the segmentation performance of the model and the accuracy of weld defect detection.

[0104] Step 5: Obtain the actual weld defect image, use the target defect detection model to detect the actual weld defect image, and obtain the weld segmentation result and the defect segmentation result.

[0105] The target defect detection model STLF-MDN of the present invention can make full use of the information resources in unlabeled data and a small amount of labeled data. When there is less labeled data, the deep features learned by DCGAN are applied to the second discriminator, and the features extracted by the second discriminator are combined with the multi-scale features of the original U-Net, which effectively solves the problems of data imbalance and scarce labeled data. Experimental results show that when the amount of labeled data is reduced to 50% of the traditional method, the present invention can still maintain a segmentation accuracy of more than 90%, significantly improving the accuracy of weld defect detection. While effectively utilizing unlabeled data and a small amount of labeled data, the present invention significantly reduces the computational complexity. STLF-MDN not only improves the detection accuracy of the model for small defects, but also maintains a high computational efficiency. In actual industrial applications, especially in scenarios with high requirements for real-time and resource utilization, it has better performance and higher application value.

[0106] Example 2

[0107] A weld safety assessment method provided by an embodiment of the present invention comprises the following steps:

[0108] Step 1: Use the weld defect detection method in Example 1 of the present application to obtain weld segmentation results and defect segmentation results.

[0109] That is, the target defect detection model in Example 1 of the present application is used to detect the weld image to be evaluated, and the weld segmentation result and defect segmentation result of the weld image are obtained.

[0110] Step 2: Determine the geometric parameters of the weld area according to the weld segmentation result; determine the number of defect areas and the geometric parameters of each defect area according to the defect segmentation result.

[0111] In a specific embodiment of the present invention, the geometric parameters of the weld area are determined according to the weld segmentation result, specifically including:

[0112] The minimum circumscribed rectangle of the weld area is determined according to the weld segmentation result, and the geometric parameters of the weld area are determined according to the minimum circumscribed rectangle of the weld area.

[0113] The weld segmentation result is the segmented weld area. In this embodiment, the geometric parameter of the weld area is the weld length, which is the longest side of the minimum circumscribed rectangle. The weld length is approximated by the longest side of the minimum circumscribed rectangle, and the overall size of the weld is accurately described in this way.

[0114] In a specific embodiment of the present invention, the geometric parameters of each defect area are determined according to the defect segmentation result, specifically including:

[0115] The minimum circumscribed figure of each defect area is determined according to the defect segmentation result, and the geometric parameters of the corresponding defect area are determined according to the minimum circumscribed figure of each defect area.

[0116] The defect segmentation result is each defect area segmented, and the shape of the minimum circumscribed figure of each defect area is determined by the shape of the defect itself. For example, the minimum circumscribed figure corresponding to the crack defect is a rectangle, and the minimum circumscribed figure corresponding to the pore defect is an ellipse. Taking the pore defect as an example, the minimum circumscribed ellipse of each pore area is determined according to the pore defect segmentation result, and the diameter of the corresponding pore area (that is, the minor axis length of the minimum circumscribed ellipse) is determined according to the minimum circumscribed ellipse of each pore area, that is, the geometric parameter of the pore area is the pore diameter.

[0117] The geometric parameters of the weld area and each defect area are converted into actual physical dimensions through a preset scaling factor, and the physical properties of the weld can be comprehensively measured through these geometric parameters.

[0118] The pixel values ​​in the segmentation results can be used to distinguish the weld area, defect area, and background area. Figure 2 As shown in (d), the pixel value of the background is 0, the pixel value of the weld area is 50, and the pixel value of the pore defect is 255. In this embodiment, the number of defective areas is greater than 0. When the number of defective areas is equal to 0, the weld has no defects and no safety evaluation is required.

[0119] Step 3: Perform weld safety assessment based on the geometric parameters of the weld area, the number of defect areas, and the geometric parameters of each defect area.

[0120] In a specific embodiment of the present invention, the weld safety evaluation is performed according to the geometric parameters of the weld area, the number of defective areas and the geometric parameters of each defective area, including:

[0121] Step 3.1: Determine a reference value according to geometric parameters of the weld area, and divide the weld area into multiple sub-areas according to the reference value;

[0122] Step 3.2: Perform weld safety evaluation of the corresponding sub-area based on the number of defective areas in each sub-area and the geometric parameters of each defective area.

[0123] Among them, the weld safety evaluation of the corresponding sub-area is performed according to the number of defective areas in each sub-area and the geometric parameters of each defective area, including:

[0124] When the geometric parameters of each defective area in the sub-area are all 0, it indicates that no defects are found in the corresponding sub-area and the weld in the corresponding sub-area is safe;

[0125] When the number of defective areas in a sub-area is less than or equal to the number threshold, and the geometric parameter of each defective area is less than or equal to the geometric parameter threshold, it indicates that although there are defects in the corresponding sub-area, the defect size meets the specified standard range, and the weld in the corresponding sub-area is safe;

[0126] When the number of defective areas in a sub-area is less than or equal to the number threshold, and the geometric parameter of each defective area is greater than the geometric parameter threshold, it indicates that the defect size of the corresponding sub-area exceeds the specified standard range, and the weld defect of the corresponding sub-area is unsafe;

[0127] When the number of defective areas in a sub-area is greater than the quantity threshold, it indicates that the number of defects in the corresponding sub-area exceeds the specified standard range, and the weld defects in the corresponding sub-area are unsafe.

[0128] According to the AWS specification, taking porosity defects as an example, the reference value of weld length is 100 mm, the number threshold is 1, and the pore diameter threshold is 2.5 mm. Within every 100 mm of weld length, the maximum diameter of the pore shall not exceed 2.5 mm, and only one visible pore is allowed, that is, the porosity shall not exceed 1. Fig. 9 shown, specifically,

[0129] When the diameter of each pore within every 100mm weld length is 0, there is no pore and it is safe;

[0130] When there is one pore in every 100mm of weld length and the pore diameter does not exceed 2.5mm (i.e. the number of defective areas ≤ the number threshold 1, and the geometric parameters of each defective area ≤ the pore diameter threshold 2.5mm), the pore diameter meets the standard and is safe;

[0131] When there is a pore in every 100mm weld length and the pore diameter exceeds 2.5mm (the number of defective areas ≤ the number threshold 1, and the geometric parameters of each defective area > the pore diameter threshold 2.5mm), it is unsafe;

[0132] When there is more than one pore in every 100mm of weld length, it is still unsafe even if the pore diameter meets the standard;

[0133] When there are more than one pore per 100mm weld length and at least one pore has a diameter greater than 2.5mm, it is unsafe.

[0134] The last two situations together mean that it is unsafe as long as the number of pores in every 100 mm of weld length exceeds one (ie the number of defective areas>the number threshold 1).

[0135] The accuracy of weld safety evaluation is improved through multi-level evaluation standards. The evaluation method of the present invention can effectively evaluate the safety of welds, help inspectors quickly identify potential defects, and ensure that the weld quality meets industrial safety standards. The present invention provides an accurate classification basis for the automated detection of weld defects, which helps to improve the reliability and efficiency of weld detection.

[0136] Fig.10 The STLF-MDN segmentation results of different weld images are shown and evaluated based on the AWS specification. Fig.10 The detected pores are shown, with the weld area marked by blue lines and the bounding boxes of the pores shown in green (≤2.5mm in diameter) or red (>2.5mm in diameter) depending on their diameter. Fig.10 It can be seen that the present invention performs well in the detection, quantification and evaluation of weld defects. Although some traditional methods can accurately identify the weld areas in almost all test images, the detection accuracy of minority classes (such as weld porosity defects) is lacking due to insufficient samples during training. However, the STLF-MDN model of the present invention significantly improves the IoU score in porosity defect detection. By adding unlabeled image data containing more minority classes (i.e., porosity defects) to train the model, the IoU score of this type of defects can be further improved, and the overall model performance can be improved. Except for a very small number of images that did not achieve the expected results, Fig.10 It is shown that all pores are accurately mapped and quantified, which meets the requirements of the AWS specification, indicating that the STLF-MDN framework of the present invention has extremely high application potential in practical industrial visual inspection applications.

[0137] Traditional methods have difficulty in accurately extracting geometric parameters (such as weld length and pore diameter) from weld images, which in turn affects the accuracy of weld safety assessment. The present invention uses automated geometric parameter extraction technology to accurately calculate weld length and pore diameter, and classifies weld images into four types according to AWS standards, including "safe" and "unsafe" situations. The evaluation method of the present invention can not only detect weld defect types, but also evaluate weld safety, ensuring the accuracy and consistency of weld defect assessment and meeting the actual needs of industrial inspection.

[0138] Through these advantages, the present invention not only significantly improves the accuracy and robustness of weld defect detection, but also greatly reduces the dependence on large-scale labeled data, and adapts to the complex and changeable weld detection needs in industrial environments.

[0139] Example 3

[0140] An embodiment of the present invention also provides an electronic device, which includes: a memory, a processor, and a computer program / instructions stored in the memory, and the processor executes the computer program / instructions to implement the weld defect detection method or weld safety assessment method in the embodiment of the present application.

[0141] Although not shown, the electronic device includes a processor, which can perform various appropriate operations and processes according to the program and / or data stored in the read-only memory (ROM) or the program and / or data loaded from the storage portion into the random access memory (RAM). The processor can be a multi-core processor, or it can include multiple processors. In some embodiments, the processor can include a general main processor and one or more special coprocessors, such as a central processing unit, a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In RAM, various programs and data required for device operation are also stored. The processor, ROM and RAM are connected to each other via a bus. The input / output (I / O) interface is also connected to the bus.

[0142] The processor and memory are used together to execute the program / instructions stored in the memory. When the program / instructions are executed by the computer, the methods, steps or functions described in the above embodiments can be implemented.

[0143] Although not shown, an embodiment of the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the weld defect detection method or weld safety assessment method in the embodiment of the present application.

[0144] Storage media in embodiments of the present invention include permanent and non-permanent, removable and non-removable items that can be implemented by any method or technology to store information. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0145] Readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0146] Although not shown, an embodiment of the present invention further provides a computer program product, including: a computer program / instruction, which, when executed by a processor, implements the weld defect detection method or weld safety assessment method in the embodiment of the present application.

[0147] What is disclosed above is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or modifications within the technical scope disclosed in the present invention, which should be covered within the protection scope of the present invention.

Claims

1. A weld defect detection method, characterized in that: The detection method comprises: Acquire a first sample data set and a second sample data set; wherein each sample in the first sample data set includes a weld image, and each sample in the second sample data set includes a weld image and its weld area and defect area; Constructing a deep convolutional generative adversarial network and a weld defect detection model; wherein the deep convolutional generative adversarial network includes a generator and a first discriminator; the weld defect detection model adopts a U-Net network, the U-Net network includes an encoder, a decoder, a newly added splicing layer, and a second discriminator having the same architecture as the first discriminator, the second discriminator is connected to the bottleneck layer in the encoder, and the newly added splicing layer is connected to the bottleneck layer, the second discriminator and the decoder in the encoder; Using the first sample data set to train a deep convolutional generative adversarial network to obtain weight parameters of a first discriminator; Transferring the weight parameters of the first discriminator to the second discriminator, and then training the U-Net network using the second sample data set to obtain a target defect detection model; An actual weld defect image is acquired, and the target defect detection model is used to detect the actual weld defect image to obtain a weld segmentation result and a defect segmentation result.

2. The weld defect detection method according to claim 1, characterized in that: The specific process of obtaining the first sample data set or the second sample data set includes: A series of weld images of the structure are collected by an industrial camera; wherein the series of weld images include normal weld images and weld defect images of different defect types; Performing a first image enhancement process on each of the weld images, and constructing a first sample data set based on the images after the first image enhancement process; Each weld image is labeled and subjected to second image enhancement processing in turn, and a second sample data set is constructed based on the image after the second image enhancement processing.

3. The weld defect detection method according to claim 2, characterized in that: The first image enhancement processing includes affine transformation, random cropping, unpaired image defogging, elastic transformation, flipping, grid distortion, perspective transformation, adding Gaussian noise, adding ISO noise, image compression, random brightness contrast, and grayscale; The second image enhancement process includes adjusting illumination, adding noise, adding random points, moving images and bounding boxes, and flipping.

4. The weld defect detection method according to any one of claims 1 to 3, characterized in that: The U-Net network is trained using the second sample data set, specifically including: Using an encoder to extract features from each weld image in the second sample data set to obtain a first feature value; Using the bottleneck layer of the encoder to perform channel compression on the first feature quantity to obtain a second feature quantity; Using the second discriminator to extract features from the second feature quantity to obtain a third feature quantity; Using the newly added splicing layer to splice the first feature quantity and the third feature quantity to obtain a fourth feature quantity; Using the decoder to extract features from the fourth feature quantity to obtain a region segmentation result; A loss value is calculated according to the region segmentation result and the corresponding weld region and defect region, and a weight parameter of the U-Net network is adjusted according to the loss value.

5. A weld safety assessment method, characterized in that: The evaluation method includes: Obtaining a weld segmentation result and a defect segmentation result using the weld defect detection method according to any one of claims 1 to 4; Determine the geometric parameters of the weld area according to the weld segmentation result; determine the number of defect areas and the geometric parameters of each defect area according to the defect segmentation result; The weld safety evaluation is performed based on the geometric parameters of the weld area, the number of defect areas and the geometric parameters of each defect area.

6. The weld safety evaluation method according to claim 5, characterized in that: Determining geometric parameters of the weld area according to the weld segmentation result specifically includes: Determine the minimum circumscribed rectangle of the weld area according to the weld segmentation result, and determine the geometric parameters of the weld area according to the minimum circumscribed rectangle of the weld area; Determining the geometric parameters of each defect area according to the defect segmentation result specifically includes: The minimum circumscribed figure of each defect area is determined according to the defect segmentation result, and the geometric parameters of the corresponding defect area are determined according to the minimum circumscribed figure of each defect area.

7. The weld safety assessment method according to claim 5 or 6, characterized in that: The weld safety evaluation is performed based on the geometric parameters of the weld area, the number of defect areas and the geometric parameters of each defect area, including: Determine a reference value according to the geometric parameters of the weld area, and divide the weld area into a plurality of sub-areas according to the reference value; Conduct weld safety evaluation of the corresponding sub-area according to the number of defective areas in each sub-area and the geometric parameters of each defective area; Among them, the weld safety evaluation of the corresponding sub-area is performed according to the number of defective areas in each sub-area and the geometric parameters of each defective area, including: When the geometric parameters of each defective area in the sub-area are 0, the weld of the corresponding sub-area is safe; When the number of defective areas in a sub-area is less than or equal to the number threshold, and the geometric parameter of each defective area is less than or equal to the geometric parameter threshold, the weld of the corresponding sub-area is safe; When the number of defective areas in a sub-area is less than or equal to the number threshold, and the geometric parameter of each defective area is greater than the geometric parameter threshold, the weld in the corresponding sub-area is unsafe; When the number of defective areas in a sub-area is greater than a quantity threshold, the weld in the corresponding sub-area is unsafe.

8. An electronic device comprising a memory, a processor, and a computer program / instruction stored in the memory, characterized in that: The processor executes the computer program / instructions to implement the weld defect detection method according to any one of claims 1 to 4 or the weld safety assessment method according to any one of claims 5 to 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the weld defect detection method according to any one of claims 1 to 4 or the weld safety evaluation method according to any one of claims 5 to 7 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the weld defect detection method according to any one of claims 1 to 4 or the weld safety evaluation method according to any one of claims 5 to 7 is implemented.

Images