Weld Defect Detection Method, Evaluation Method, Equipment, Storage Medium and Product

Abstract

The present invention discloses a method for detecting weld defects, an evaluation method, a device, a storage medium and a product. The detection method includes constructing a deep convolutional generative adversarial network and a weld defect detection model. Among them, the deep convolutional generative adversarial network includes a generator and a first discriminator. The weld defect detection model adopts a U-Net network and includes an encoder, a decoder, a newly added splicing layer and a second discriminator having the same architecture as the first discriminator. The deep convolutional generative adversarial network is trained using a first sample dataset to obtain the weight parameters of the first discriminator. The weight parameters of the first discriminator are migrated to the second discriminator, and then the U-Net network is trained using a second sample dataset to obtain a target defect detection model. The target defect detection model is used to detect an actual weld defect image to obtain a weld segmentation result and a defect segmentation result. The present invention reduces the dependence on labeled data while improving the detection accuracy of weld defects.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a weld defect detection method, evaluation method, device, storage medium and product based on semi-supervised transfer learning and feature fusion. Background Art

[0002] In industrial production, the integrity of welds directly affects the structural safety of products. Defects in welds (such as pores, cracks, etc.) may pose serious safety hazards if not detected and processed in time. Current weld defect detection technologies mainly rely on manual inspection and automated inspection methods based on image processing. However, manual inspection is inefficient and easily affected by the subjective judgment of inspectors, resulting in inconsistent inspection results. In addition, traditional image processing technologies have performance bottlenecks when dealing with complex backgrounds and subtle defects in weld images, and it is difficult to adapt to the data imbalance situation in the actual industrial environment.

[0003] In recent years, with the rapid development of deep learning technologies, automated weld defect detection methods based on convolutional neural networks (CNNs) have gradually become the mainstream. Deep learning models such as CNNs can automatically extract features from images. Especially in the semantic segmentation task of weld images, such as the application of the U-Net network architecture, certain results have been achieved. However, deep learning models generally rely on a large amount of labeled data for training, and in the industrial weld inspection scenario, the acquisition of labeled data is often very difficult and time-consuming. Therefore, in the case of insufficient labeled data, existing deep learning models are prone to overfitting, affecting the generalization ability of the models, especially when dealing with different types of weld defects.

[0004] In addition, the defects in welds are usually small in size and unevenly distributed, with a large gap in the pixel ratio compared to the weld body. This data imbalance problem leads to weak detection ability of deep learning models in small-sized defect areas, and it is easy to miss detections or make false detections. In addition, there are various types of weld defects, including pores, cracks, lack of fusion, etc. Traditional deep learning models are difficult to maintain high detection accuracy in weld images with complex backgrounds and different defect morphologies. Although data augmentation techniques can alleviate the problem of insufficient data to a certain extent, it is still a great challenge to select appropriate data augmentation methods.

[0005] In summary, the existing weld defect detection solutions, especially in complex industrial environments, still have the following significant deficiencies:

[0006] (1) Strong dependence on labeled data: Existing deep learning models require a large amount of labeled data for training. In the case of scarce labeled data, the generalization ability of the models is insufficient, and overfitting is prone to occur; while in industrial scenarios, it is time-consuming and laborious to label large-scale data, and the acquisition process is extremely difficult.

[0007] (2) Data imbalance problem: Weld defects are usually small-sized and rare, with a serious imbalance in pixel distribution compared to the weld background, resulting in low detection accuracy of the deep learning model for defect areas and prone to missed detections and false detections.

[0008] (3) Diversity and complexity of defect detection: There are various types of weld defects (including pores, cracks, etc.) with different shapes. Especially in a complex industrial background, traditional detection methods are difficult to adapt to these diverse defect shapes and lack robustness.

[0009] Although existing research has attempted to improve the performance of weld image semantic segmentation by improving the deep learning architecture (such as the U-Net network), these methods still perform poorly in dealing with data imbalance, small samples, and complex backgrounds. Especially in the detection of small defects, the accuracy and robustness of existing methods are still not ideal. Therefore, how to improve the accuracy and efficiency of weld defect detection under the conditions of limited labeled data, uneven defect distribution, and complex background has become an urgent problem to be solved. Summary of the Invention

[0010] The purpose of the present invention is to provide a weld defect detection method, evaluation method, device, storage medium, and product to solve at least one of the problems that the traditional weld defect detection technology based on deep learning models is highly dependent on labeled data, the data imbalance of weld defects leads to low detection accuracy of traditional methods, and the diversity and complexity of weld defect detection lead to poor robustness of traditional methods.

[0011] The present invention solves the above technical problems through the following technical solutions: A weld defect detection method includes:

[0012] Obtain a first sample data set and a second sample data set; wherein, each sample in the first sample data set includes a weld image, and each sample in the second sample data set includes a weld image, its weld area, and defect area;

[0013] Construct a deep convolutional generative adversarial network and a weld defect detection model; wherein, the deep convolutional generative adversarial network includes a generator and a first discriminator; the weld defect detection model uses a U-Net network, and the U-Net network includes an encoder, a decoder, an additional splicing layer, and a second discriminator having the same architecture as the first discriminator. The second discriminator is connected to the bottleneck layer in the encoder, and the additional splicing layer is connected to the bottleneck layer, the second discriminator, and the decoder in the encoder;

[0014] Use the first sample data set to train the deep convolutional generative adversarial network to obtain the weight parameters of the first discriminator;

[0015] Transfer the weight parameters of the first discriminator to the second discriminator, and then use the second sample dataset to train the U-Net network to obtain a target defect detection model;

[0016] Obtain an actual weld defect image, and use the target defect detection model to detect the actual weld defect image to obtain a weld segmentation result and a defect segmentation result.

[0017] Further, the specific process of obtaining the first sample dataset or the second sample dataset includes:

[0018] Collect a series of weld images of the structure through an industrial camera; among them, a series of weld images include normal weld images and weld defect images of different defect types;

[0019] Perform first image enhancement processing on each weld image, and construct a first sample dataset according to the image after the first image enhancement processing;

[0020] Perform annotation and second image enhancement processing on each weld image in sequence, and construct a second sample dataset according to the image after the second image enhancement processing.

[0021] Further, the first image enhancement processing includes affine transformation, random cropping, non-paired image dehazing, elastic transformation, flipping, grid distortion, perspective transformation, adding Gaussian noise, adding ISO noise, image compression, random brightness contrast, grayscale conversion;

[0022] The second image enhancement processing includes illuminance adjustment, adding noise, adding random points, moving the image and the bounding box, flipping.

[0023] Further, using the second sample dataset to train the U-Net network specifically includes:

[0024] Use the encoder to extract features from each weld image in the second sample dataset to obtain a first feature quantity;

[0025] Use the bottleneck layer of the encoder to compress the channels of the first feature quantity to obtain a second feature quantity;

[0026] Use the second discriminator to extract features from the second feature quantity to obtain a third feature quantity;

[0027] Use the new splicing layer to splice the first feature quantity and the third feature quantity to obtain a fourth feature quantity;

[0028] Use the decoder to extract features from the fourth feature quantity to obtain a region segmentation result;

[0029] Calculate the loss value according to the region segmentation result and the corresponding weld region and defect region, and adjust the weight parameters of the U-Net network according to the loss value.

[0030] Based on the same concept, the present invention provides a weld safety evaluation method, including:

[0031] Obtain the weld segmentation result and the defect segmentation result by using the weld defect detection method as described above;

[0032] Determine the geometric parameters of the weld region according to the weld segmentation result; determine the number of defect regions and the geometric parameters of each defect region according to the defect segmentation result;

[0033] Conduct a weld safety evaluation according to the geometric parameters of the weld region, the number of defect regions, and the geometric parameters of each defect region.

[0034] Further, determining the geometric parameters of the weld region according to the weld segmentation result specifically includes:

[0035] Determine the minimum circumscribed rectangle of the weld region according to the weld segmentation result, and determine the geometric parameters of the weld region according to the minimum circumscribed rectangle of the weld region;

[0036] Determining the geometric parameters of each defect region according to the defect segmentation result specifically includes:

[0037] Determine the minimum circumscribed figure of each defect region according to the defect segmentation result, and determine the geometric parameters of the corresponding defect region according to the minimum circumscribed figure of each defect region.

[0038] Further, conducting a weld safety evaluation according to the geometric parameters of the weld region, the number of defect regions, and the geometric parameters of each defect region includes:

[0039] Determine a reference value according to the geometric parameters of the weld region, and divide the weld region into multiple sub-regions according to the reference value;

[0040] Conduct a weld safety evaluation of the corresponding sub-region according to the number of defect regions in each sub-region and the geometric parameters of each defect region;

[0041] Among them, conducting a weld safety evaluation of the corresponding sub-region according to the number of defect regions in each sub-region and the geometric parameters of each defect region includes:

[0042] When the geometric parameters of each defect region in the sub-region are all 0, the weld of the corresponding sub-region is safe;

[0043] When the number of defect regions in the sub-region ≤ the number threshold, and the geometric parameter of each defect region ≤ the geometric parameter threshold, the weld of the corresponding sub-region is safe;

[0044] When the number of defect regions in a sub-region ≤ the number threshold and the geometric parameter of each defect region > the geometric parameter threshold, the weld of the corresponding sub-region is unsafe;

[0045] When the number of defect regions in a sub-region > the number threshold, the weld of the corresponding sub-region is unsafe.

[0046] Based on the same concept, the present invention further provides an electronic device, including a memory, a processor, and a computer program / instructions stored on the memory, and the processor executes the computer program / instructions to implement the weld defect detection method or the weld safety evaluation method as described above.

[0047] Based on the same concept, the present invention further provides a computer-readable storage medium, on which a computer program / instructions is stored, and when the computer program / instructions is executed by a processor, the weld defect detection method or the weld safety evaluation method as described above is implemented.

[0048] Based on the same concept, the present invention further provides a computer program product, including a computer program / instructions, and when the computer program / instructions is executed by a processor, the weld defect detection method or the weld safety evaluation method as described above is implemented.

[0049] Advantageous Effects

[0050] Compared with the prior art, the advantages of the present invention are as follows:

[0051] A weld defect detection method provided by the present invention learns deep features from a large amount of unlabeled data through a deep convolutional generative adversarial network, and uses transfer learning to transfer the discriminator weight parameters obtained by the deep convolutional generative adversarial network to the discriminator of the U-Net network, which reduces the dependence on labeled data while improving the defect detection accuracy, and effectively alleviates the problems of insufficient labeled data and difficulty in obtaining; the present invention uses the newly added discriminator in the U-Net network to further extract features from the features extracted by the encoder, and splices the further extracted features with the features extracted by the encoder through the newly added splicing layer, and then decodes by the decoder, which enhances the learning of complex features in the weld image by the U-Net network, is more conducive to the detection of diverse defects, complex defects and / or subtle defects, and improves the weld defect detection accuracy and detection robustness.

[0052] A weld safety evaluation method provided by the present invention evaluates the weld safety according to the geometric parameters of the weld region, the number of defect regions and the geometric parameters of each defect region, solves the problem of data imbalance caused by small and unevenly distributed weld defects, and improves the accuracy of weld safety evaluation. Description of the Drawings

[0053] To more clearly illustrate the technical solution of the present invention, the accompanying drawings required for the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only one embodiment of the present invention. For those of ordinary skill in the art, other accompanying drawings can be obtained based on these drawings without creative efforts.

[0054] Figure 1 is the flowchart of the weld defect detection method in the embodiment of the present invention;

[0055] Figure 2 is the weld image in the embodiment of the present invention; among them, (a) is the original weld image collected by the industrial camera, (b) is the cropped weld image, (c) is the labeled weld image, and (d) is the grayscale weld image;

[0056] Figure 3 is the architecture diagram of the deep convolutional generative adversarial network in the embodiment of the present invention;

[0057] Figure 4 is the flowchart of the multi-domain network training based on semi-supervised transfer learning and feature fusion in the embodiment of the present invention;

[0058] Figure 5 is the original U-Net network architecture diagram in the embodiment of the present invention;

[0059] Figure 6 is the comparison flowchart of the target defect detection model and the classical U-Net network in the embodiment of the present invention;

[0060] Figure 7 is the Dice score comparison curve of the STLF-MDN, the original U-Net, and the AUnet three models in the embodiment of the present invention;

[0061] Figure 8 is the IoU score comparison curve of the STLF-MDN, the original U-Net, and the AUnet three models in the embodiment of the present invention;

[0062] Figure 9 is the weld safety evaluation process when there are pore defects in the embodiment of the present invention;

[0063] Figure 10 is the STLF-MDN segmentation and evaluation results of different weld images in the embodiment of the present invention. Detailed implementation manners

[0064] Combined with the accompanying drawings in the embodiments of the present invention, the technical solutions in the present invention are clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative work belong to the scope of protection of the present invention.

[0065] The following specifically describes the technical solutions of the present application with specific embodiments. These several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0066] Existing weld defect detection technologies, especially deep learning methods based on convolutional neural networks (CNNs), show significant deficiencies in dealing with high-noise backgrounds and small defects. First, these methods highly rely on large-scale labeled datasets. In industrial weld detection, it is difficult to obtain labeled data, resulting in the model being prone to overfitting during the training process and a significant decline in detection accuracy. Second, they lack sufficient flexibility in dealing with the irregular shapes and different sizes of weld defects, and it is difficult to effectively capture multi-scale defect features, especially being particularly weak in the detection of small-sized defects. Finally, in scenarios with data imbalance, traditional deep learning models often ignore small-sized defects, leading to a high miss detection rate. At the same time, these methods require cumbersome parameter adjustment and a long training cycle, and it is difficult to meet the requirements of real-time and high efficiency in industrial applications. Therefore, there are still obvious deficiencies in the existing technologies in terms of accuracy, speed, and adaptability, especially in weld detection applications with strict requirements such as steel bridge decks, where their limitations are more prominent. Based on this, the present invention provides a weld defect detection method and evaluation method based on semi-supervised transfer learning and feature fusion.

[0067] Embodiment 1

[0068] As Figure 1 shown, a weld defect detection method provided by an embodiment of the present invention includes the following steps:

[0069] Step 1: Obtain a first sample dataset and a second sample dataset.

[0070] In the specific implementation manner of the present invention, the specific acquisition process of the first sample dataset or the second sample dataset includes:

[0071] Step 1.1: Collect a series of weld images of the structure through an industrial camera; among them, a series of weld images include normal weld images and weld defect images of different defect types;

[0072] Step 1.2: Perform first image enhancement processing on each weld image, and construct a first sample dataset according to the image after the first image enhancement processing;

[0073] Step 1.3: Sequentially perform annotation and second image enhancement processing on each weld image, and construct a second sample dataset based on the image after the second image enhancement processing.

[0074] Each sample in the first sample dataset includes a weld image, and each sample in the second sample dataset includes a weld image and its weld area and defect area. That is, the first sample dataset is an unannotated dataset, and the second sample dataset is an annotated dataset. In this embodiment, the first image enhancement processing includes affine transformation, random cropping, non-paired image defogging (such as the 4D method), elastic transformation, flipping, grid distortion, perspective transformation, adding Gaussian noise, adding ISO noise, image compression, random brightness contrast, grayscale conversion; the second image enhancement processing includes illuminance adjustment, adding noise, adding random points, moving the image and bounding box, flipping.

[0075] To solve the problem of strong dependence on the annotated dataset, the present invention obtains two datasets, one is an unannotated dataset and the other is an annotated dataset. The weight file generated by learning the features of the unannotated dataset is transferred to the weld defect detection model through transfer learning, which can improve the weld defect detection accuracy while reducing the dependence on the annotated dataset. To solve the problem of data scarcity, the present invention performs first image enhancement processing and second image enhancement processing on each weld image collected by an industrial camera respectively. Through region-level and pixel-level image enhancement techniques, the diversity of data is extended while ensuring that the mask remains unchanged, effectively alleviating the problem of data scarcity.

[0076] Exemplarily, a series of weld images of a steel bridge deck are obtained using an industrial camera. The image shooting distance is about 20 cm, and the resolution of the original image collected is 4000×1846 pixels, as shown in (a) of Figure 2 ; these images are then cropped to a size of 1024×1024 pixels (as shown in (b) of Figure 2 ), and different regions are manually annotated with different colors, as shown in (c) of Figure 2 ; then, the annotated color image is converted into a grayscale label image and divided into three categories pixel by pixel: a pixel value of 0 represents the background, 50 represents the weld area, and 255 represents the pore category, as shown in (d) of Figure 2 . The types of the original weld images collected are shown in Table 1, and the weld defect types include pores.

[0077] Table 1 Weld Image Types

[0078]

[0079] The first sample dataset and the second sample dataset generated after the first image enhancement process and the second image enhancement process are shown in Table 2 and Table 3 respectively.

[0080] Table 2 First Sample Dataset

[0081]

[0082] Table 3 Second Sample Dataset

[0083]

[0084] Step 2: Construct a deep convolutional generative adversarial network and a weld defect detection model.

[0085] As Figure 3 shown, the deep convolutional generative adversarial network (DCGAN model) includes a generator and a first discriminator. The weld defect detection model adopts a U-Net network, which includes an encoder, a decoder, a newly added splicing layer, and a second discriminator with the same architecture as the first discriminator. The second discriminator is connected to the bottleneck layer in the encoder, and the newly added splicing layer is connected to the bottleneck layer, the second discriminator, and the decoder in the encoder, as Figure 4 shown. The U-Net network of the present invention is a variant of the original U-Net network (as Figure 5 shown), that is, on the basis of the original U-Net network, a second discriminator and a splicing layer are added to the bottleneck layer of the encoder. The second discriminator is used to further extract features from the features output by the bottleneck layer, and the newly added splicing layer is used to splice (i.e., fuse) the features output by the bottleneck layer and the features output by the second discriminator, and then the decoder decodes the spliced features. Figure 4 The multi-domain network based on semi-supervised transfer learning and feature fusion in

[0086] refers to a network obtained by unsupervised training of the deep convolutional generative adversarial network using the first sample dataset to obtain the weight parameters of the first discriminator, migrating the weight parameters of the first discriminator to the second discriminator of the U-Net network through transfer learning, and finally performing supervised training on the U-Net network using the second sample dataset.

[0087] Through unsupervised training, the deep convolutional generative adversarial network extracts deep semantic features from the unlabeled dataset (i.e., the first sample dataset), including edge, texture, and shape features in the weld image. These features are reflected in the weight parameters of the first discriminator. Migrating the weight parameters of the first discriminator to the second discriminator can improve the ability of the U-Net network to capture image details at different scales, further improving the semantic segmentation accuracy and the generalization ability of the U-Net network.

[0088] Step 4: Transfer the weight parameters of the first discriminator to the second discriminator, and then use the second sample dataset to train the U-Net network to obtain the target defect detection model.

[0089] In a specific embodiment of the present invention, training the U-Net network using the second sample dataset specifically includes:

[0090] Step 4.1: Use the encoder to extract features from each weld image in the second sample dataset to obtain the first feature quantity;

[0091] Step 4.2: Use the bottleneck layer of the encoder to compress the channels of the first feature quantity to obtain the second feature quantity;

[0092] Step 4.3: Use the second discriminator to extract features from the second feature quantity to obtain the third feature quantity;

[0093] Step 4.4: Use the newly added splicing layer to splice the first feature quantity and the third feature quantity to obtain the fourth feature quantity;

[0094] Step 4.5: Use the decoder to extract features from the fourth feature quantity to obtain the region segmentation result;

[0095] Step 4.6: Calculate the loss value according to the region segmentation result and the corresponding weld region and defect region, and adjust the weight parameters of the U-Net network according to the loss value.

[0096] The bottleneck layer of the encoder compresses the channels of the input 64×64×1024 high-dimensional feature map (i.e., the first feature quantity) through 1×1 convolution operation, reducing the number of channels to 64×64×3 (i.e., the second feature quantity) to ensure that the second feature quantity output by the bottleneck layer is compatible with the input size of the second discriminator. This channel compression strategy not only reduces the computational complexity but also ensures the transmission of key information by selectively retaining high-information channels. The second feature quantity is input into the second discriminator, and the second discriminator uses its hierarchical convolution structure to gradually reduce the dimension and extract high-order features from the second feature quantity, finally generating a 4×4×512 high-level feature map (i.e., the third feature quantity). These high-level feature maps contain high-level semantic information after multiple convolutions and have strong class discrimination ability and fine structure recognition ability.

[0097] To effectively fuse these high-level feature maps (i.e., the third feature quantity) with the multi-scale features output by the encoder (i.e., the first feature quantity), the present invention uses the bilinear interpolation method to spatially up-sample the 4×4×512 high-level feature map output by the second discriminator, restoring it to a resolution of 64×64×512. Through the bilinear interpolation method, the high-order semantic information is extended to a scale matching the original feature map, ensuring spatial consistency between the feature maps. After obtaining the same spatial dimension, the feature map with a resolution of 64×64×512 is concatenated with the 64×64×1024 high-dimensional feature map originally output by the bottleneck layer in the channel dimension to form a concatenated feature map of 64×64×1536 (i.e., the fourth feature quantity). By concatenation, the model can flexibly capture weld defects of different scales and forms, significantly improving the detection accuracy and robustness, especially showing a significant improvement in the detection of small defects. To avoid the introduction of redundant information and at the same time enhance the expression ability of the features, a 1×1 convolutional kernel is used to perform depth convolution operations on the concatenated feature map, compressing its number of channels to 1024 to match the input requirements of the decoder.

[0098] The target defect detection model (i.e., the semi-supervised transfer learning and feature fusion-based multi-domain network STLF-MDN (Semi-Supervised Transfer Learning-Based Multi-Domain Network with Feature Fusion)) can still achieve efficient weld defect detection under the condition of limited labeled data, especially performing well in scenarios dealing with small-sized defects and complex backgrounds, avoiding the overfitting problem and improving the generalization ability of the model.

[0099] To verify the robustness and accuracy of the target defect detection model, the segmentation results of the target defect detection model are compared with the segmentation results of multiple classic U-Net networks, as Figure 6 shown. By comparing the key performance indicators of each model (such as IoU (Intersection over Union) and Dice score), the best segmentation model is determined to be the target defect detection model of the present invention.

[0100] Five comparison models and the target defect detection model of the present invention were all trained after applying the "early stopping" callback to obtain the best Dice score and IoU score. The results are shown in Table 4. For the target defect detection model of the present invention, before training, the weight parameters extracted from the first discriminator (DCGAN was trained on the first sample dataset and the callback was applied, and the loss of the first discriminator was reduced to 0.05) were transferred to the second discriminator of the U-Net network of the present invention; subsequently, the U-Net network of the present invention was trained in the second sample dataset. According to the test IoU score, three best segmentation models were finally selected for further analysis, namely the target defect detection model STLF-MDN of the present invention, the original U-Net, and AUnet (Attention Unet).

[0101] Table 4 Comparison of performance indicators between the target defect detection model of the present invention and the comparison models

[0102] Serial number Model IoU score Dice score Number of training epochs 1 STLF-MDN 93.3 0.917 174 2 Original Unet 89.4 0.881 161 3 AUnet 90.7 0.899 225 4 U-Net++ 83.8 0.882 158 5 ResUNet 84.1 0.884 162 6 R2U-Net 81.1 0.875 142

[0103] Due to the adoption of the early stopping strategy, the number of training rounds of each model is different. To fairly compare the segmentation performance of each model, the index data at the 160th training round was selected, and the performance of the three models of STLF-MDN, the original U-Net, and AUnet was further analyzed. The results are as Figure 7 and Figure 8 shown. Compared with the U-Net and AUnet models, the Dice scores of STLF-MDN increased by 4.6% and 4.7% respectively, and the IoU scores increased by 9.1% and 11.2% respectively; in addition, the original U-Net had the fastest training speed when reaching the IoU score of 90. From the Dice curve ( Figure 7 ), it can be seen that the STLF-MDN model captured effective features faster than other models during the training process, thus reaching a higher performance level at an earlier stage, and the curve also tended to be stable earlier. It shows that the STLF-MDN model of the present invention has greatly improved the segmentation performance of the model and the accuracy of weld defect detection.

[0104] Step 5: Obtain the actual weld defect image, and use the target defect detection model to detect the actual weld defect image to obtain the weld segmentation result and the defect segmentation result.

[0105] The target defect detection model STLF-MDN of the present invention can make full use of the information resources in unlabeled data and a small amount of labeled data. In the case of less labeled data, the deep features learned by DCGAN are applied to the second discriminator, and the features extracted by the second discriminator are combined with the multi-scale features of the original U-Net, effectively solving the problems of data imbalance and scarce labeled data. The experimental results show that when the amount of labeled data is reduced to 50% of the traditional method, the present invention can still maintain a segmentation accuracy of more than 90%, significantly improving the accuracy of weld defect detection. While effectively using unlabeled data and a small amount of labeled data, the present invention significantly reduces the computational complexity. STLF-MDN not only improves the detection accuracy of the model for small defects, but also maintains high computational efficiency, and has better performance and higher application value in actual industrial applications, especially in scenarios with high requirements for real-time performance and resource utilization.

[0106] Embodiment 2

[0107] A weld safety evaluation method provided by an embodiment of the present invention includes the following steps:

[0108] Step 1: Obtain the weld segmentation result and the defect segmentation result by using the weld defect detection method in Embodiment 1 of the present application.

[0109] That is, use the target defect detection model in Embodiment 1 of the present application to detect the weld image to be evaluated, and obtain the weld segmentation result and the defect segmentation result of the weld image.

[0110] Step 2: Determine the geometric parameters of the weld area according to the weld segmentation result; determine the number of defect areas and the geometric parameters of each defect area according to the defect segmentation result.

[0111] In a specific embodiment of the present invention, determining the geometric parameters of the weld area according to the weld segmentation result specifically includes:

[0112] Determine the minimum circumscribed rectangle of the weld area according to the weld segmentation result, and determine the geometric parameters of the weld area according to the minimum circumscribed rectangle of the weld area.

[0113] The weld segmentation result is the segmented weld area. In this embodiment, the geometric parameter of the weld area is the weld length, and the weld length is the longest side of the minimum circumscribed rectangle. The length of the weld is approximately represented by the longest side of the minimum circumscribed rectangle, and the overall size of the weld is accurately described in this way.

[0114] In a specific embodiment of the present invention, determining the geometric parameters of each defect area according to the defect segmentation result specifically includes:

[0115] Determine the minimum circumscribed figure of each defect area according to the defect segmentation result, and determine the geometric parameters of the corresponding defect area according to the minimum circumscribed figure of each defect area.

[0116] The defect segmentation result is each defect area segmented out. The shape of the minimum circumscribed figure of each defect area is determined by the shape of the defect itself. For example, the minimum circumscribed figure corresponding to a crack defect is a rectangle, and the minimum circumscribed figure corresponding to a pore defect is an ellipse. Taking the pore defect as an example, determine the minimum circumscribed ellipse of each pore area according to the pore defect segmentation result, and determine the diameter of the corresponding pore area (i.e., the length of the minor axis of the minimum circumscribed ellipse) according to the minimum circumscribed ellipse of each pore area. That is, the geometric parameter of the pore area is the pore diameter.

[0117] Convert the geometric parameters of the weld area and the geometric parameters of each defect area into actual physical dimensions through a preset scaling factor. Through these geometric parameters, the physical characteristics of the weld can be comprehensively measured.

[0118] The weld area, defect area, and background area can be distinguished by the pixel values in the segmentation result. As Figure 2 shown in (d) of, the pixel value of the background is 0, the pixel value of the weld area is 50, and the pixel value of the pore defect is 255. In this embodiment, when the number of defect areas is greater than 0, when the number of defect areas is equal to 0, the weld has no defects and no safety evaluation is required.

[0119] Step 3: Conduct a safety evaluation of the weld according to the geometric parameters of the weld area, the number of defect areas, and the geometric parameters of each defect area.

[0120] In the specific embodiment of the present invention, the safety evaluation of the weld is conducted according to the geometric parameters of the weld area, the number of defect areas, and the geometric parameters of each defect area, including:

[0121] Step 3.1: Determine a reference value according to the geometric parameters of the weld area, and divide the weld area into multiple sub-areas according to the reference value;

[0122] Step 3.2: Conduct a safety evaluation of the weld in the corresponding sub-area according to the number of defect areas in each sub-area and the geometric parameters of each defect area.

[0123] Among them, conducting a safety evaluation of the weld in the corresponding sub-area according to the number of defect areas in each sub-area and the geometric parameters of each defect area includes:

[0124] When the geometric parameters of each defect area in the sub-area are all 0, it indicates that no defects are found in the corresponding sub-area, and the weld in the corresponding sub-area is safe;

[0125] When the number of defect areas within the sub-region ≤ the quantity threshold and the geometric parameters of each defect area ≤ the geometric parameter threshold, it indicates that although there are defects in the corresponding sub-region, the defect size meets the specified standard range and the weld of the corresponding sub-region is safe;

[0126] When the number of defect areas within the sub-region ≤ the quantity threshold and the geometric parameters of each defect area > the geometric parameter threshold, it indicates that the defect size in the corresponding sub-region exceeds the specified standard range and the weld of the corresponding sub-region is not safe;

[0127] When the number of defect areas within the sub-region > the quantity threshold, it indicates that the number of defects in the corresponding sub-region exceeds the specified standard range and the weld of the corresponding sub-region is not safe.

[0128] According to the specification of the American Welding Society AWS, taking pore defects as an example, the reference value of the weld length is 100 mm, the quantity threshold is 1, the pore diameter threshold is 2.5 mm. Within every 100 mm of weld length, the maximum diameter of the pores shall not exceed 2.5 mm and only one visible pore is allowed, that is, the porosity does not exceed 1. As Figure 9 shown, specifically,

[0129] When the diameter of each pore is 0 within every 100 mm of weld length, there are no pores and it is safe;

[0130] When there is one pore within every 100 mm of weld length and the pore diameter does not exceed 2.5 mm (that is, the number of defect areas ≤ the quantity threshold 1 and the geometric parameters of each defect area ≤ the pore diameter threshold 2.5 mm), the pore diameter meets the standard and it is safe;

[0131] When there is one pore within every 100 mm of weld length and the pore diameter exceeds 2.5 mm (the number of defect areas ≤ the quantity threshold 1 and the geometric parameters of each defect area > the pore diameter threshold 2.5 mm), it is not safe;

[0132] When the number of pores within every 100 mm of weld length exceeds one, even if the pore diameter meets the standard, it is still not safe;

[0133] When the number of pores within every 100 mm of weld length exceeds one and at least one pore diameter exceeds 2.5 mm, it is not safe.

[0134] The last two cases combined mean that as long as the number of pores within every 100 mm of weld length exceeds one (that is, the number of defect areas > the quantity threshold 1), it is not safe.

[0135] The accuracy of weld safety evaluation is improved through multi-level evaluation criteria. The evaluation method of the present invention can effectively evaluate the safety of welds, help inspectors quickly identify potential defect problems, ensure that the weld quality meets industrial safety standards, and provides an accurate classification basis for the automated detection of weld defects, contributing to improving the reliability and efficiency of weld inspection.

[0136] Figure 10 The STLF-MDN segmentation results of different weld images are shown and evaluated based on AWS specifications. Figure 10 The detected pores are shown. The weld area is marked with blue lines, and the bounding boxes of the pores are shown in green (diameter ≤ 2.5 mm) or red (diameter > 2.5 mm) according to their diameters. Figure 10 It can be seen that the present invention performs excellently in the detection, quantification, and evaluation of weld defects. Although some traditional methods can almost accurately identify the weld areas in all test images, in the detection of a few types (such as weld pore defects), due to insufficient samples during training, the detection accuracy is lacking. However, the STLF-MDN model of the present invention significantly improves the IoU score in the detection of pore defects. By training the model with additional unlabeled image data containing more of the minority class (i.e., pore defects), the IoU score for this type of defect can be further improved, and the overall model performance can be enhanced. Except for a very small number of images that do not meet the expected results, Figure 10 all the pores shown are accurately mapped and quantified, meeting the requirements of AWS specifications, indicating that the STLF-MDN framework of the present invention has extremely high application potential in actual industrial vision inspection applications.

[0137] Traditional methods have difficulty in accurately extracting geometric parameters (such as weld length and pore diameter) from weld images, thereby affecting the accuracy of weld safety evaluation. The present invention can accurately calculate the weld length and pore diameter through automated geometric parameter extraction technology, and classify weld images into four types according to AWS standards, including various situations of "safe" and "unsafe". The evaluation method of the present invention can not only detect the types of weld defects but also evaluate the safety of welds, ensuring the accuracy and consistency of weld defect evaluation and meeting the actual needs of industrial inspection.

[0138] With these advantages, the present invention not only significantly improves the accuracy and robustness of weld defect detection but also greatly reduces the dependence on large-scale labeled data, adapting to the complex and variable weld inspection requirements in industrial environments.

[0139] Example 3

[0140] An embodiment of the present invention also provides an electronic device, which includes: a memory, a processor, and a computer program / instructions stored in the memory, and the processor executes the computer program / instructions to implement the weld defect detection method or the weld safety evaluation method in the embodiments of the present application.

[0141] Although not shown, the electronic device includes a processor that can perform various appropriate operations and processes according to the programs and / or data stored in the read-only memory (ROM) and / or the programs and / or data loaded from the storage section into the random access memory (RAM). The processor can be a multi-core processor or can include multiple processors. In some embodiments, the processor can include a general main processor and one or more special coprocessors, such as a central processing unit, a graphics processing unit (GPU), a neural network processing unit (NPU), a digital signal processor (DSP), and so on. In the RAM, various programs and data required for the operation of the device are also stored. The processor, ROM, and RAM are connected to each other through a bus. The input / output (I / O) interface is also connected to the bus.

[0142] The above-mentioned processor and the memory are jointly used to execute the program / instructions stored in the memory, and when the program / instructions are executed by a computer, they can implement the methods, steps, or functions described in the above-mentioned various embodiments.

[0143] Although not shown, an embodiment of the present invention also provides a computer-readable storage medium, on which a computer program / instructions are stored, and when the computer program / instructions are executed by a processor, they implement the weld defect detection method or the weld safety evaluation method in the embodiments of the present application.

[0144] In the embodiments of the present invention, the storage medium includes permanent and non-permanent, removable and non-removable articles that can implement information storage by any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0145] A readable storage medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media do not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0146] Although not shown, an embodiment of the present invention also provides a computer program product, including: a computer program / instructions, which when executed by a processor, implement the weld defect detection method or the weld safety evaluation method in the embodiments of the present application.

[0147] The above-disclosed are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or variations, which should all be covered within the protection scope of the present invention.

Claims

1. A method for detecting weld defects, characterized in that, The detection method includes: Obtaining a first sample data set and a second sample data set; wherein, each sample in the first sample data set includes a weld image, and each sample in the second sample data set includes a weld image, its weld area, and defect area; Constructing a deep convolutional generative adversarial network and a weld defect detection model; wherein, the deep convolutional generative adversarial network includes a generator and a first discriminator; the weld defect detection model uses a U-Net network, and the U-Net network includes an encoder, a decoder, a newly added splicing layer, and a second discriminator having the same architecture as the first discriminator. The second discriminator is connected to the bottleneck layer in the encoder, and the newly added splicing layer is connected to the bottleneck layer, the second discriminator, and the decoder in the encoder; Training the deep convolutional generative adversarial network using the first sample data set to obtain the weight parameters of the first discriminator; Transferring the weight parameters of the first discriminator to the second discriminator, and then training the U-Net network using the second sample data set to obtain a target defect detection model; Obtaining an actual weld defect image, and using the target defect detection model to detect the actual weld defect image to obtain a weld segmentation result and a defect segmentation result; Among them, training the U-Net network using the second sample data set specifically includes: Using the encoder to perform feature extraction on each weld image in the second sample data set to obtain a first feature quantity; Using the bottleneck layer of the encoder to perform channel compression on the first feature quantity to obtain a second feature quantity; Using the second discriminator to perform feature extraction on the second feature quantity to obtain a third feature quantity; Using the newly added splicing layer to splice the first feature quantity and the third feature quantity to obtain a fourth feature quantity; Using the decoder to perform feature extraction on the fourth feature quantity to obtain a region segmentation result; Calculating a loss value according to the region segmentation result and the corresponding weld area and defect area, and adjusting the weight parameters of the U-Net network according to the loss value.

2. The weld defect detection method according to claim 1, wherein, The specific process of obtaining the first sample data set or the second sample data set includes: Collecting a series of weld images of the structure through an industrial camera; wherein, a series of weld images include normal weld images and weld defect images of different defect types; Performing a first image enhancement process on each weld image, and constructing a first sample data set according to the image after the first image enhancement process; Performing annotation and a second image enhancement process on each weld image in sequence, and constructing a second sample data set according to the image after the second image enhancement process.

3. The weld defect detection method according to claim 2, wherein The first image enhancement process includes affine transformation, random cropping, unpaired image dehazing, elastic transformation, flipping, grid distortion, perspective transformation, adding Gaussian noise, adding ISO noise, image compression, random brightness contrast, and grayscale conversion; The second image enhancement process includes illuminance adjustment, adding noise, adding random points, moving the image and the bounding box, and flipping.

4. A weld seam safety evaluation method, characterized in that, The evaluation method includes: Obtain the weld seam segmentation result and the defect segmentation result by using the weld seam defect detection method described in any one of claims 1 to 3; Determine the geometric parameters of the weld seam area according to the weld seam segmentation result; determine the number of defect areas and the geometric parameters of each defect area according to the defect segmentation result; Conduct a weld seam safety assessment according to the geometric parameters of the weld seam area, the number of defect areas, and the geometric parameters of each defect area.

5. The weld seam safety evaluation method according to claim 4, characterized in that, Determine the geometric parameters of the weld seam area according to the weld seam segmentation result, specifically including: Determine the minimum circumscribed rectangle of the weld seam area according to the weld seam segmentation result, and determine the geometric parameters of the weld seam area according to the minimum circumscribed rectangle of the weld seam area; Determine the geometric parameters of each defect area according to the defect segmentation result, specifically including: Determine the minimum circumscribed figure of each defect area according to the defect segmentation result, and determine the geometric parameters of the corresponding defect area according to the minimum circumscribed figure of each defect area.

6. The weld seam safety evaluation method according to claim 4 or 5, characterized in that, Conduct a weld seam safety assessment according to the geometric parameters of the weld seam area, the number of defect areas, and the geometric parameters of each defect area, including: Determine a reference value according to the geometric parameters of the weld seam area, and divide the weld seam area into multiple sub-areas according to the reference value; Conduct a weld seam safety assessment of the corresponding sub-area according to the number of defect areas in each sub-area and the geometric parameters of each defect area; Among them, conducting a weld seam safety assessment of the corresponding sub-area according to the number of defect areas in each sub-area and the geometric parameters of each defect area includes: When the geometric parameters of each defect area in the sub-area are all 0, the weld seam of the corresponding sub-area is safe; When the number of defect areas in the sub-area ≤ the number threshold, and the geometric parameters of each defect area ≤ the geometric parameter threshold, the weld seam of the corresponding sub-area is safe; When the number of defect areas in the sub-area ≤ the number threshold, and the geometric parameters of each defect area > the geometric parameter threshold, the weld seam of the corresponding sub-area is unsafe; When the number of defect areas in the sub-area > the number threshold, the weld seam of the corresponding sub-area is unsafe.

7. An electronic device, comprising a memory, a processor, and a computer program / instructions stored on the memory, characterized in that, The processor executes the computer program / instructions to implement the weld seam defect detection method described in any one of claims 1 to 3 or the weld seam safety assessment method described in any one of claims 4 to 6.

8. A computer-readable storage medium having computer programs / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the weld seam defect detection method described in any one of claims 1 to 3 or the weld seam safety assessment method described in any one of claims 4 to 6.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the weld seam defect detection method described in any one of claims 1 to 3 or the weld seam safety assessment method described in any one of claims 4 to 6.

Images