Industrial weld defect detection method based on semantic segmentation

Abstract

The application discloses an industrial weld defect detection method based on semantic segmentation, comprising the following steps: S1, acquiring original weld images for training and related parameter information, carrying out binaryzation processing on the original images, storing the foreground area of the processed images, selecting the largest area for rectangular shape conversion, cutting the rectangular area and saving; S2, labeling the cut rectangular area images, generating a weld defect image dataset, cutting the images in the weld defect image dataset using an intelligent cutting method, and distributing the excess pixels generated by the cutting; S3, constructing a semantic segmentation network model according to the dataset generated in S2, focusing on the defect area using an attention mechanism, selectively carrying out convolution calculation on the defect area, and using a loss function as an index to train the semantic segmentation network model; S4, processing the weld images to be detected using the semantic segmentation network model, and splicing and restoring to the original picture size. The application improves the detection precision and the image processing is not limited.

Description

Technical Field

[0001] This invention relates to the field of welding inspection technology, and in particular to a method for detecting defects in industrial welds based on semantic segmentation. Background Technology

[0002] Industrial welding technology is widely used in many key fields such as aviation, aerospace, nuclear energy, chemical industry, shipbuilding, electronics, construction, transportation, power, and machinery manufacturing, and is an indispensable part of modern industry. However, due to various factors, welds or welded joints often exhibit defects that do not meet design or standard requirements, specifically manifesting as imperfections, discontinuities, or harmful conditions. These defects not only weaken the strength and stability of the welded structure but may also pose a potential threat to the performance and safety of the welded components. Currently, although various welding quality inspection technologies exist, they are often limited by inspection accuracy, efficiency, or cost. Therefore, accurate and efficient inspection of welding quality remains an indispensable and urgent issue to be addressed in ensuring industrial safety.

[0003] Currently, in industrial radiographic weld defect inspection, traditional methods rely on manual visual inspection, which limits accuracy, especially in identifying minute defects (such as micro-cracks and porosity). This directly affects the safety and reliability of welded structures. With the rapid development of modern industrial production, the number of welds has increased dramatically. Traditional inspection methods are time-consuming and inefficient, making it difficult to meet the inspection needs of large-scale production. Manual inspection not only requires a large amount of human resources and high training costs, but also struggles to guarantee the stability and consistency of inspection results, further increasing overall costs. Furthermore, manual inspection is susceptible to factors such as the inspector's experience, fatigue, and concentration, leading to a high rate of misjudgments and missed detections, posing a potential threat to product quality and safety. Summary of the Invention

[0004] To address the issues of low detection accuracy and limited image processing in existing weld defect detection technologies, this invention proposes an industrial weld defect detection method based on semantic segmentation.

[0005] The specific technical solution is as follows:

[0006] A semantic segmentation-based method for detecting defects in industrial welds includes:

[0007] S1: Obtain the original weld seam image and related parameter information for training, perform binarization processing on the original image, store the foreground region of the processed image, select the region with the largest area for rectangular shape transformation, crop the rectangular region and save it;

[0008] S2: Annotate the cropped rectangular region image, generate a weld defect image dataset, use an intelligent cutting method to cut the images in the weld defect image dataset, and allocate the excess pixels generated by the cutting.

[0009] S3: Construct a semantic segmentation network model based on the dataset generated in S2, use an attention mechanism to focus on defective regions, perform selective convolution calculations on defective regions, and use a loss function as an evaluation metric to train the semantic segmentation network model;

[0010] S4: Use a semantic segmentation network model to process the weld seam image under test and stitch it back to the original image size.

[0011] Furthermore, intelligent cutting methods include:

[0012] A1: Obtain the input image parameters, determine the width and height of the input image parameters, and if the width is less than the height, rotate the input image by 90 degrees so that the width is always the larger parameter.

[0013] A2: Determine the number of image segments to be divided. Obtain the average width of the image based on the number of segments. Determine the ratio of width to height. If the ratio of width to height is less than 2, adjust the image size to 256*256. If the ratio of width to height is greater than or equal to 2, then cut the original image according to the number of image segments, use the average width of the image as the width of all images, save the size and number of the cut images, and adjust the size of the cut images to 256*256.

[0014] Furthermore, methods for allocating redundant pixels include:

[0015] B1: Determine the basic width of the image and identify the number of redundant pixels. The basic width is calculated by integer division to determine the width value to be allocated to each segment, and the number of redundant pixels is determined by modulo operation.

[0016] B2: Traverse each segment of the image and determine the relationship between the index of the current segment and the number of redundant pixels;

[0017] B3: Determine whether to increase the width of an image by one pixel based on the index relationship.

[0018] Furthermore, semantic segmentation network models include:

[0019] Image segmentation network: The architecture is based on Transformer and U-Net, which is suitable for small object detection and is used for feature extraction and transmission;

[0020] RRC module: used for lightweight convolution, further fusing information from different layers after decoupling between convolutional layers;

[0021] The CRA module is used to enhance the spatial correlation and context capture capabilities of the model when processing image semantic segmentation tasks. It divides the input feature map into windows, generates multiple weight matrices for the divided regions, performs feature projection, calculates the correlation between different regions based on the weight matrices, and selects regions based on the correlation between different regions.

[0022] Furthermore, the loss function is a combination of specific weights used to guide the model in accurately learning the boundaries of each category, addressing the class imbalance problem, and enhancing the ability to detect defects in small targets. The loss function is expressed as:

[0023] Total Loss = λ CE ·CE Loss+λ Dice ·Dice Loss+λ Focal Focal Loss

[0024] Where, λ CE , λ Dice , λ Focal These are the weight coefficients for cross-entropy loss, Dice loss, and focus loss, respectively.

[0025] Furthermore, S1 also includes:

[0026] S11: Perform an expansion operation on the rectangular region, with the expansion size set to 21*21 pixels. Calculate the boundary coordinates of the minimum bounding rectangle of the expansion region. The boundary coordinates include the row positions of the upper and lower boundaries of the rectangle.

[0027] S12: Store the boundary coordinate information into the corresponding variables, generate a new rectangular region based on the calculated boundary coordinates of the minimum bounding rectangle, crop the processed original image based on the generated rectangular region, and save the cropped image.

[0028] Furthermore, the original image processing in S1 also includes: connecting the binarized foreground regions, merging adjacent regions into a whole, and storing the merged region into a variable.

[0029] Furthermore, S2 also includes: for each segment, determining the start and end positions of the cropping region based on its allocated width and the current left boundary; after cropping a segment, updating the current left boundary to the right boundary of the previous cropping region.

[0030] Furthermore, the processing of the dataset in S2 also includes: augmenting the dataset using random flipping and random rotation, and adding the augmented data to the dataset.

[0031] Furthermore, S4 also includes:

[0032] S41: Input the image of the weld to be tested, use the semantic segmentation network model to cut the image, and record the size data of each original image of the weld to be tested and the information of the same image;

[0033] S42: Perform contrast stretching, adaptive histogram equalization, and gamma correction sequentially on the cut weld seam image to enhance the image;

[0034] S43: Input the enhanced image from S42 into the semantic segmentation network model and output the prediction result;

[0035] S44: Based on the image number of the predicted cut, restore each slice to the position corresponding to the original weld seam image to be tested, and seamlessly restore the image to its original size.

[0036] The above technical solution has the following advantages or technical effects:

[0037] 1. The cutting method used in this invention is not limited by the image size of 256. Instead of directly cutting the portion of the image that exceeds 256 in size at least twice, the method controls the size of the image to around 256 after cutting, which greatly reduces computational redundancy and improves segmentation efficiency.

[0038] 2. When dealing with situations where the image width cannot be divided evenly by the expected number of slices, this invention ensures that the size of all sliced ​​images remains consistent by reasonably allocating pixel values, avoiding abrupt or inconsistent size changes. Resizing the image afterwards will greatly reduce the possibility of distortion and make the results more accurate.

[0039] 3. In the construction of the semantic segmentation network model of the present invention, the detection accuracy of the present invention is significantly improved by introducing RRC and CRA technologies. Compared with the previous network model, its performance is superior. In addition, the model adopts a semantic segmentation algorithm, which achieves pixel-level detailed detection compared with the previous target detection method, thus achieving significant progress in detection accuracy and detail capture.

[0040] 4. In constructing the loss function, this invention introduces focus loss and assigns weights. The final optimized overall loss function significantly improves the overall model's detection performance and ability to detect small target defects. The loss function, which combines specific weights, serves as the model's evaluation index, thereby improving the model's accuracy in learning the boundary recognition of different types of weld seams and solving the model's ability to cope with class imbalance problems.

[0041] 5. Compared with traditional single image enhancement techniques, contrast stretching, histogram equalization and gamma correction, although each has its unique advantages, often fail to fully meet the needs of complex image scenes. This invention combines contrast stretching and adaptive histogram equalization, and uses gamma correction, resulting in better image enhancement effects. Attached Figure Description

[0042] Figure 1 This is a flowchart of the method of the present invention;

[0043] Figure 2 This is a flowchart of the image cropping process of the present invention;

[0044] Figure 3 This is the original, uncropped image of the present invention;

[0045] Figure 4 This is a cropped image of the present invention;

[0046] Figure 5 This is a flowchart of the image cutting process of the present invention;

[0047] Figure 6 This is a network structure diagram of the present invention;

[0048] Figure 7 This is a structural diagram of the Transformer Block of the present invention;

[0049] Figure 8 This is a structural diagram of the RRC module of the present invention;

[0050] Figure 9 This is a structural diagram of the CRA module of the present invention;

[0051] Figure 10 This is a flowchart of the image enhancement process of the present invention;

[0052] Figure 11 These are image effect comparison diagrams of the present invention. Detailed Implementation

[0053] To make the technical solution of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] like Figure 1 As shown, the industrial weld defect detection method based on semantic segmentation includes:

[0055] S1: Obtain the original weld seam image and related parameter information for training, perform binarization processing on the original image, store the foreground region of the processed image, select the region with the largest area for rectangular shape transformation, crop the rectangular region and save it;

[0056] S2: Annotate the cropped rectangular region image, generate a weld defect image dataset, use an intelligent cutting method to cut the images in the weld defect image dataset, and allocate the excess pixels generated by the cutting.

[0057] S3: Construct a semantic segmentation network model based on the dataset generated in S2, use an attention mechanism to focus on defective regions, perform selective convolution calculations on defective regions, and use a loss function as an evaluation metric to train the semantic segmentation network model;

[0058] S4: Use a semantic segmentation network model to process the weld seam image under test and stitch it back to the original image size.

[0059] This invention integrates traditional visual technology with artificial intelligence deep learning methods, proposing an image processing and defect recognition method. This method can automatically and efficiently process X-ray images and accurately identify various welding defects, including copper / tungsten inclusions, cracks, lack of fusion, incomplete penetration, and slag / porosity, through algorithms. The implementation of this invention will effectively overcome the difficulties of manual evaluation, reduce the inspection costs in the welding industry, and improve inspection speed and accuracy. It is applicable to various welding types and materials, and can work stably and reliably in various industrial environments, providing strong technical support for improving welding quality and ensuring industrial safety.

[0060] The original weld images contain image numbers and some manually added annotations, which are redundant information for weld defect detection tasks and need to be removed through preprocessing. For example... Figure 2 As shown, the present invention provides a process for cropping weld seam images.

[0061] First, the image file in the specified path is read to obtain its size information, including its width and height. This size information will be used in subsequent image processing operations to ensure that the processed area conforms to the actual image size.

[0062] Next, the image is binarized, and the pixels with values ​​between 60 and 255 are identified as foreground regions. The processed result is stored. This step separates the foreground region from the background, facilitating subsequent processing.

[0063] Then, these binarized foreground regions are connected, merging adjacent regions into a single entity. These merged regions are stored in variables. This step ensures that all connected regions are correctly identified and processed.

[0064] From all connected regions, select the region with the largest area and use it as the most important foreground region, then store the result. This ensures that the processing focus area is maximized.

[0065] The selected largest area is transformed into a rectangle. This step helps define a clear processing area, facilitating further operations.

[0066] The rectangular region is dilated to expand its boundaries, with the dilation size set to 21x21 pixels. This step includes any details that might have been missed and provides sufficient coverage for subsequent image cropping.

[0067] Calculate the boundary coordinates of the minimum bounding rectangle of the expansion region, including the row positions of the rectangle's upper and lower boundaries. This boundary coordinate information is stored in corresponding variables, providing the basis for generating the clipping region.

[0068] Based on the calculated boundary coordinates of the minimum bounding rectangle, a new rectangular region is generated to define the cropping area. This generated rectangular region is then used to crop the original image. Finally, the cropped image is saved. Image cropping removes redundant information from the original weld image, improving before-and-after comparison. Figure 3 , Figure 4 As shown.

[0069] The LabelMe tagging tool was used to create corresponding pixel-level defect image annotations, forming a weld defect image dataset. The dataset includes various defect types such as cracks, porosity, slag inclusions, incomplete penetration, and incomplete fusion. In practical applications, images captured from industrial environments often exhibit non-standardized size characteristics, with the image width significantly larger than its height, for example, commonly ranging from 4209x1418 pixels to 3920x648 pixels. For such input images, if their resolution is significantly higher than the preset processing standard, traditional scaling methods will inevitably lead to a significant loss of key information in the image, thus adversely affecting subsequent image detection and analysis processes, specifically manifesting as a significant decrease in detection accuracy.

[0070] This invention prioritizes the width dimension and proposes an intelligent image segmentation method based on the image's aspect ratio (width-to-height ratio) to segment the preprocessed image, such as... Figure 5 As shown, it specifically includes:

[0071] Obtain the input image parameters, determine the width W and height H of the input image parameters, and if the width W is less than the height H, rotate the input image by 90 degrees so that the width is always the larger parameter.

[0072] The number of image segments, `split_pic_num`, is determined according to the segmentation rules. This number is related to the ratio `ar` of width (W) and height (H), where `ar = W / H`. The original image width is divided by the number of segments, `split_pic_num`, to obtain the average width (`average_width`). The ratio of width (W) to height (H) is then compared. If the ratio is less than 2, the image is resized to 256*256. If the ratio is greater than or equal to 2, the original image is segmented according to the number of segments, resulting in several equal parts. The average width of each segment is used as the width of all segments. The dimensions and numbers of the segmented images are saved, and the resized images are resized to 256*256. The results are then re-input into the model testing task.

[0073] The division rules are as follows:

[0074] 1.0<=ar<1.5---->split_pic_num=1

[0075] 1.5<=ar<2.5---->split_pic_num=2

[0076] 2.5<=ar<3.5---->split_pic_num=3

[0077] 3.5<=ar<4.5---->split_pic_num=4

[0078] 4.5<=ar<5.5---->split_pic_num=5

[0079] 5.5<=ar<6.5---->split_pic_num=6

[0080] 6.5<=ar<7.5---->split_pic_num=7

[0081] 7.5<=ar<8.5---->split_pic_num=8 . . .

[0085] 14.5<=ar<=15.5---->split_pic_num=15

[0086] The intelligent cropping algorithm in this invention can effectively preserve key image information while flexibly adjusting input images of different resolutions, thereby ensuring improved detection accuracy and overall system performance while maintaining image information integrity. Existing technologies typically simply crop all images to a size of 256*256 pixels, filling any portions with a height less than 256 pixels with blank space. When the height is less than 256 pixels, the number of cropped images is N; when the image height is greater than 256 pixels, the number of cropped images is 2N. This is equivalent to inspecting an original image, doubling the time and resources required. However, the processing method of this invention does not simply crop all images to a size of 256*256 pixels. After the initial image cropping process in the weld seam image cropping process, the image height is already within a suitable range. Using an intelligent cutting algorithm, preprocessing is performed with a size close to 256 pixels, followed by interpolation processing. This is not limited by the image height of 256 pixels, and the final number of cut images is all N. After cutting, the image size is controlled to be close to 256 pixels. Resizing the image will greatly reduce the possibility of distortion, making the result more accurate. The processing result is more natural and effective than the existing technology.

[0087] This invention proposes an intelligent image segmentation method based on image aspect ratio. This method can automatically adapt to images with different aspect ratios. By accurately calculating and dynamically adjusting the segmentation strategy, it ensures that the image is effectively adjusted to the size required by the network input while maintaining the integrity of key image information, thereby improving the overall processing efficiency and detection accuracy.

[0088] After intelligent segmentation using segmentation rules, in order to avoid the problem that image pixels cannot be divided evenly, this invention also proposes a special pixel allocation mechanism and cropping mechanism. This method aims to deal with the situation where the image width cannot be divided evenly by the expected number of segments. By reasonably allocating pixel values, it ensures that the size of all segmented images remains consistent and avoids abrupt or inconsistent size changes.

[0089] The mechanisms for allocating redundant pixels include:

[0090] The image's base width (part_width) and the number of extra pixels (extra_pixels) are determined. The base width is calculated using integer division (width / / split_parts) to ensure that the width of each part remains consistent when extra pixels are ignored. The number of extra pixels (extra_pixels) is determined using modulo operation (width%split_parts) to determine the number of extra pixels (extra_pixels) that cannot be evenly distributed. These pixels will be distributed according to specific rules in subsequent steps.

[0091] Iterate through each segment of the image and consider the relationship between the index of the current segment and the number of redundant pixels.

[0092] The decision to increase the width of an image by one pixel is based on the index relationship. If the index of the current part is less than the number of extra pixels (extra_pixels), the width of that part is increased by one pixel to ensure that the extra pixels are evenly distributed among the first extra_pixels parts in sequence.

[0093] The pruning mechanism includes:

[0094] Cropping region calculation: For each segment, the start and end positions of the cropping region are determined based on its allocated width (including any possible increase of one pixel) and the current left boundary (current_left). The cropping region starts from current_left and ends at current_left + part_width_with_extra (including any increase in pixels, if applicable).

[0095] Boundary update: After completing the cropping of a part, update the current left boundary (current_left) to the right boundary of the previous cropping region (i.e., current_left + part_width_with_extra) to ensure that the cropping of subsequent parts starts from the correct position.

[0096] For ease of understanding, the present invention provides the following examples:

[0097] For images whose width is not divisible by a small number of extra pixels, set the image width to 605 pixels and divide it into 3 parts.

[0098] Basic width calculation: 605 / / 3 = 201, the basic width of each part is 201 pixels.

[0099] Redundant pixel calculation: 605%3 = 2, there are 2 redundant pixels.

[0100] Redundant pixel allocation: Add 1 pixel to the first two parts.

[0101] Part 1: 201 + 1 = 202 pixels;

[0102] Part 2: 201 + 1 = 202 pixels;

[0103] Part 3: 201 pixels;

[0104] Segmentation result: The three parts are 202, 202, and 201 pixels respectively, with the last part being one pixel short.

[0105] Conclusion: When the image width cannot be divided evenly, the mechanism intelligently distributes the excess pixels evenly to the first few parts, avoiding large uneven distribution.

[0106] The above implementation process effectively solves the problem of redundant pixels caused by uneven pixel distribution. The intelligent allocation strategy ensures the rationality of the size of each segment. Finally, by adjusting the aspect ratio of the image to a ratio close to 1, the image data is optimized to better suit the needs of subsequent model training, thereby improving the effect and efficiency of model processing.

[0107] After completing the above process, the dataset is augmented using random flipping and random rotation, and the augmented data is added to the dataset. This completes the creation of the dataset. The created dataset will then be fed into the network below.

[0108] Most existing weld defect detection schemes focus on target-level detection of defects using target detection algorithms. While these algorithms can achieve detection, their accuracy is lower than that of semantic segmentation. Furthermore, in practical applications, defect size is crucial. Therefore, the semantic segmentation-based detection method proposed in this invention allows for accurate determination of defect size directly from the high-precision results obtained after detection, facilitating subsequent defect level assessment.

[0109] like Figure 6 As shown, this invention proposes a semantic segmentation neural network model, RC-WeldUnet, based on ResNet50 and Transformer blocks. This model utilizes an image segmentation network with a Transformer and U-Net architecture to adapt to small object detection. This design aims to fully leverage the advantages of Transformer in handling global dependencies and combine it with U-Net's capabilities in handling local features and efficient upsampling. Skip connections directly pass high-resolution features from the encoder to the decoder, contributing to improved segmentation accuracy. In the overall structure, fast and efficient spatial feature extraction is achieved using a CNN in the encoder section, and the feature maps are then passed to the Transformer layer for further global information integration. Furthermore, an RRC module is used to further reduce the number of parameters and achieve redundant computation. Choose Region Attention is used before the Transformer Block; this attention mechanism allows the model to focus on target defect regions when processing images. The Transformer Block structure is as follows: Figure 7 As shown. In this way, the model can more accurately identify and segment key parts of an image.

[0110] The following table shows some of the parameters involved in the model:

[0111] Table 1: Network Parameter Table

[0112]

[0113]

[0114] A semantic segmentation network model, RC-WeldUnet, was constructed. During the construction process, RRC and CRA technologies were introduced, which significantly improved the detection accuracy of this invention (measured by mIoU). The details are explained below.

[0115] In weld defect detection, a self-attention mechanism is used, which incurs significant computational costs while obtaining a global perception. To improve model performance, a combination of self-attention and convolution is employed. However, since traditional convolution greatly increases memory access frequency, this invention considers using a lightweight convolution method. Although PConv can reduce some Flops, this invention still believes there is room for further redundancy reduction and proposes a lightweight RRConv module (RRC module), short for Reducedundancyconvolution.

[0116] The RRConv feature extraction module functions as a novel convolutional module within the network, primarily serving to further fuse information from different layers after inter-layer decoupling. This invention will be further elaborated below (e.g.) Figure 8 The diagram illustrates how to further optimize cost when dealing with feature map redundancy. Because feature maps across different channels have high similarity, this invention retains a portion of the feature map information and performs convolution operations only on a subset of the feature maps.

[0117] The RRC module's input data is I. RRC ∈R H*W*C After inputting the data, the feature map is divided into two branches, and the segmentation ratio r in this paper is... a The value is set from an initial 1.0 to a final 0.25. This is based on the splitting ratio r. a Divide channel C into p and 1-C p Channels 1-C p ∈R H*W*(1-Cp) This is the identity mapping branch, which does not perform any operations on the data, preserving the original image information to alleviate the gradient vanishing problem in deep networks and promote better information flow and model training. The remaining channels C... p ∈R H*W*Cp For the feature extraction branch, then use C p Feature extraction is performed using k*k convolutional kernels. Memory access complexity is H*W*C.p 2 *K 2 This avoids redundant calculations caused by similar feature maps, achieving a good balance between accuracy and compression.

[0118] The current number of Flops is still relatively large, not meeting expectations. Inspired by DWConv, this invention is based on the previous C p Each channel is considered representative of the entire feature map and calculated to become C p Each channel is computed on the feature map using independent k*k convolutions. This method further reduces FLOPS to H*W*K. 2 *C p Flops reduce C p times.

[0119] After completing the above operations, the results of the two branches are concatenated, thereby enhancing the information flow between different layers and better preserving and extracting the original information. Finally, a 1*1 convolution is used to fuse all features to obtain O∈R. H *W*C The final Flops become: H*W*K 2 *C p +H*W*C 2 .

[0120] This invention deals with small target defect detection tasks, so most of the content consists of irrelevant elements such as background. To address this, this invention designs a region selection routing mechanism, the CRA (ChooseRegionAttention) module, which is a sparse attention mechanism. Figure 9 As shown, this is to enhance the model's spatial relevance and context capture capabilities when processing image semantic segmentation tasks, thereby achieving efficient and accurate image segmentation.

[0121] For the input feature map, F∈R H*W*C After being partitioned by the Transformer window, the image is divided into N distinct non-overlapping sub-regions, each representing an independent processing unit. For each sub-region, a query (Q), key (K), and value (V) are generated. This is achieved by applying three different weight matrices W to each sub-region of the original feature map F. Q W K W V This is achieved through multiplication. These weight matrices are learnable parameters that define a linear transformation for each type of projection.

[0122] Q = F'W q

[0123] K = F'W K

[0124] V = F'W V

[0125] Where F' is the reorganized feature block, which needs to be adjusted in some dimensions to fit the weight matrix. W q W K W V ∈R C*C ,QKV∈R N2*HW / N2*C .

[0126] Building upon the above, the average characteristics of each region are then calculated to generate the region-level query Q. i and K j ∈R N2 *C .

[0127] Q i =avg(Q), K j =avg(k), where i and j represent different regions.

[0128] Use the generated Q i and K j To obtain the degree of relevance between different regions, this invention will use Q... i and K j D is obtained through matrix multiplication. ij ∈R N2*N2 The formula is as follows:

[0129] D ij =Q i ·(K j )T, where Q i K is the query vector for region i. j It is the key vector of region j.

[0130] The adjacency matrix describes the strength of the relationship between each region and all other regions. This representation considers not only the internal characteristics of each region but also its association with other regions.

[0131] Finally, all the D ij The values ​​are stored in the adjoining matrix D∈R N*N*N2*N2 In this context, the first two dimensions, N*N, represent all possible combinations of ij, which constitute a location region index. The latter two dimensions, N... 2 *N 2 Represents each D ij The size of the tensor is N*N*N. However, this tensor is not intuitive, so it is reconstructed. The thought tensor D has dimensions N*N*N. 2*N 2 Based on the information in D, the dimension can be transformed into (N*N)*(N 2 *N 2 The two-dimensional matrix facilitates the subsequent region selection strategy of this invention.

[0132] The most relevant regions are selected based on their relevance; each element of D is selected through the query vector Q. i and bond vector K j The dot product of D reflects the similarity or matching degree between regions in the feature space. D contains N. 2 D ij All D ij Each row represents the interaction strength between that particular region and all other regions in the image.

[0133] This invention further employs a top-n strategy to refine the main regions that should be focused on in each region, by selecting D ij The largest n values ​​in each row can be used to construct a region index matrix R. i Each row in this matrix contains indices of other regions that should be prioritized for the corresponding region. This step is achieved through the following operation:

[0134] R i =N(D[i,j])

[0135] Where i,j∈R N*N The role of the N function is to select the indices of the largest n elements from the adjacency matrix D[i,j], ensuring that the attention mechanism can focus on the most relevant regional features.

[0136] Furthermore, the model employs a semantic segmentation algorithm, specifically a loss function, which achieves pixel-level detailed detection compared to previous object detection methods, thus making significant progress in detection accuracy and detail capture.

[0137] In the semantic segmentation of weld defects, traditional single loss functions are often insufficient to comprehensively address all challenges. While cross-entropy loss can effectively guide the model to learn class boundaries, it falls short in handling class imbalance problems; Dice loss can enhance the detection ability of small targets, but lacks accurate learning of other class boundaries; focus loss focuses on solving class imbalance and hard-to-classify samples, but when used alone, it may not fully utilize the model's global learning ability.

[0138] To address this, this invention proposes a combined loss function mechanism. This mechanism combines the three aforementioned loss functions with specific weights, serving as an evaluation metric to guide model parameter updates and collectively impacting the model training process. This combined loss function not only guides the model to accurately learn the boundaries of each category but also effectively addresses class imbalance and particularly enhances the detection capability for defects in small targets.

[0139] The specific formula is as follows:

[0140] Total Loss = λ CE ·CE Loss+λ Dice ·Dice Loss+λ Focal Focal Loss

[0141] Where, λ CE , λ Dice , λ Focal These are the weight coefficients for cross-entropy loss, Dice loss, and focus loss, respectively.

[0142] Cross-entropy loss, a classic choice for classification tasks, has the core advantage of effectively measuring the difference between the model's predicted probability distribution and the true label distribution. In this model, for a six-class classification problem (including background), cross-entropy loss not only ensures the model's accurate learning of the boundaries of each class, but also supports an efficient gradient descent optimization process through its good differentiability.

[0143] The specific formula is as follows:

[0144]

[0145] Where N is the total number of samples, C is the total number of categories (including background), and y nc For the true label (0 or 1), p nc Predict probabilities for the model.

[0146] Given that the weld defect area is relatively small compared to the background area, and the class imbalance problem is significant, this invention introduces Dice coefficient loss to enhance the model's ability to detect small target defects. Dice loss directly optimizes the overlap between the predicted region and the real defect region, which is particularly effective in handling the class imbalance problem in semantic segmentation.

[0147] The formula is:

[0148]

[0149] Among them, Y xy The image is a binary image with the true label. is the probability graph predicted by the model (after sigmoid transformation), and ò is a smoothing term to prevent the denominator from being zero.

[0150] To address the issue of model over-focusing on easily classified samples due to class imbalance, this invention further integrates Focal Loss. This loss function dynamically adjusts the loss weights for samples of different difficulty levels, enabling the model to focus more on samples that are difficult to classify, thereby improving the overall model's generalization ability and robustness.

[0151] The specific form of focus loss is:

[0152]

[0153] Where, α t Used to balance positive and negative samples, pt is the model's predicted probability for the target class, and γ is the focusing parameter that adjusts the weight of easily classified samples.

[0154] This invention compares the detection accuracy (in terms of mIoU) for different combinations of loss functions:

[0155] Table 2: Comparison of Loss Function Effects

[0156]

[0157] Compared to other semantic segmentation models, the RC-WeldUnet model used in this invention has a significant advantage in detection accuracy (measured by mIoU):

[0158] Table 3: Comparison of Experimental Results of Semantic Segmentation Models

[0159]

[0160]

[0161] At this point, the entire model training phase is complete. Next, the trained model will be used for weld seam detection. During the weld seam detection phase, image segmentation processing must be performed on each input image. This processing is based on an intelligent segmentation method, which requires additional recording of the size data of each original image and information about the same image. This recording ensures that segmented image fragments from different sources are clearly distinguished and identified. Therefore, during subsequent image fusion, these recorded information can be used to accurately reconstruct the segmented fragments into the original image, thereby supporting comprehensive analysis and evaluation of weld seam quality. The specific steps are as follows:

[0162] Input the image of the weld to be tested, use a semantic segmentation network model to segment the image, and record the size data of each original image of the weld to be tested and the information of the same image.

[0163] The cut weld seam image is sequentially subjected to contrast stretching, adaptive histogram equalization, and gamma correction for image enhancement, such as... Figure 10 As shown, it specifically includes:

[0164] Apply contrast stretching to the image. This step enhances detail by adjusting the image's brightness and contrast, making low-brightness and high-brightness areas of the image more visible. Specifically, this involves calculating the low and high percentile values ​​of the image's brightness (e.g., 2% and 98%), and then remapping the image pixel values ​​based on these values ​​to expand the brightness range and increase contrast. For example, the OpenCV function `cv2.convertScaleAbs()` can be used to ensure that the image data type is suitable for contrast stretching.

[0165] Adaptive Histogram Equalization (CLAHE) is applied to the contrast-stretched image. CLAHE enhances image contrast through local equalization, particularly in areas of low contrast. This process involves converting the image from the BGR color space to the YUV color space, applying the CLAHE algorithm to the Y channel (luminance channel), and finally converting the image back to the BGR color space. CLAHE parameters (such as clipLimit = 1.0 and tileGridSize = (8,8)) control the intensity of equalization and the size of local processing, thereby improving local image contrast while avoiding over-equalization.

[0166] Finally, gamma correction is performed on the image. Gamma correction is used to adjust the brightness of the image to optimize its visual effect or adapt it to specific display needs. A lookup table (LUT) is created by calculating the inverse of the gamma value (1.0 / gamma) and applied to the image data to adjust the image brightness. Gamma correction helps correct any uneven brightness that may have been introduced in the previous processing steps and ensures that the image brightness distribution is suitable for practical use.

[0167] The image enhancement methods described above show poor results with contrast stretching and adaptive histogram equalization alone. However, combining these enhancement methods with gamma correction yields better results. These image enhancement methods improve test performance, such as... Figure 11 The images are shown for comparison. A is the original image; B is the result of using histogram equalization alone; C is the result of using contrast enhancement alone; and D is the result of image enhancement following the above process.

[0168] The enhanced image is input into the semantic segmentation network model RC-WeldUnet, and the prediction results are as follows:

[0169] Table 4: Model Test Results

[0170] Class Background track nofu nope cir tun IoU 0.9984 0.8222 0.8125 0.8024 0.8662 0.9776 Precision 0.9977 0.8480 0.9325 0.8625 0.8649 0.9200 Recall 0.9979 0.8264 0.9319 0.8631 0.8432 0.9449 F1 score 0.9978 0.8371 0.9322 0.8628 0.8539 0.9323

[0171] Based on the predictions made by the deep learning model, the final step is to stitch and restore the predicted and segmented images. This involves adjusting the images from the uniform 256x256 size used in model processing back to the actual size of the original images, ensuring seamless fusion of parts from the same image. According to the image numbering of the predicted and segmented images, each slice is restored to its corresponding position in the original weld seam image, seamlessly restoring the image to its original size. The image fusion process requires handling the following:

[0172] Table 5: Corresponding processing methods for image fusion

[0173]

[0174] Through image fusion technology, we can reconstruct multiple 256x256 pixel slices generated by the model to their original image dimensions using bicubic interpolation, based on the original slice number and size information of each slice. Then, we fuse them according to their numbers, ensuring seamless reconstruction of the entire image. During this process, we pay particular attention to properly handling overlapping areas between slices to guarantee the consistency and accuracy of classification results, effectively avoiding misclassifications that may result from improper stitching.

[0175] After completing the entire weld inspection process, the inspection results we obtained not only match the original image size, but also provide pixel-level defect size information due to the use of semantic segmentation algorithms, further improving the accuracy and practicality of the inspection.

[0176] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for detecting defects in industrial welds based on semantic segmentation, characterized in that, include: S1: Obtain the original weld seam image and image parameter information for training, perform binarization processing on the original image, store the foreground region of the processed image, select the region with the largest area for rectangular shape transformation, crop the rectangular region and save it; S2: Annotate the cropped rectangular region image, generate a weld defect image dataset, use an intelligent cutting method to cut the images in the weld defect image dataset, and allocate the excess pixels generated by the cutting. The intelligent cutting method includes: A1: Obtain the input image parameters, determine the width and height of the input image parameters, and if the width is less than the height, rotate the input image by 90 degrees so that the width is always the larger parameter. A2: Determine the number of image segments to be divided, obtain the average width of the image based on the number of segments, and determine the ratio of width to height. If the ratio of width to height is less than 2, adjust the image size to 256*256. If the ratio of width to height is greater than or equal to 2, then cut the original image according to the number of image segments, use the average width of the image as the width of all images, save the size and number of the cut images, and adjust the size of the cut images to 256*256. S3: Construct a semantic segmentation network model based on the dataset generated in S2, use an attention mechanism to focus on defective regions, perform selective convolution calculations on defective regions, and use a loss function as an evaluation metric to train the semantic segmentation network model; The semantic segmentation network model includes: Image segmentation network: The architecture is based on Transformer and U-Net, which is suitable for small object detection and is used for feature extraction and transmission; RRC module: used for lightweight convolution, further fusing information from different layers after decoupling between convolutional layers; The CRA module is used to enhance the spatial correlation and context capture capabilities of the model when processing image semantic segmentation tasks. It divides the input feature map into windows, generates query Q, key K and value V from the divided regions and performs feature projection. It also calculates the correlation degree between different regions based on query Q, key K and value V, and selects regions based on the correlation degree between the different regions. The input data for the RRC module is I RRC ∈R H*W*C After inputting the data, the feature map is divided into two branches according to the segmentation ratio r. a Divide channel C into C p and 1-C p Channel 1-C p ∈R H*W*（1-Cp） This is the identity mapping branch, which does not perform any operations on the data, preserving the original image information to alleviate the gradient vanishing problem in deep networks and promote better information flow and model training; the remaining channels C p ∈R H*W*Cp For the feature extraction branch, then use C p The features are extracted using k*k convolutional kernels; where H represents the height of the input image and W represents the width of the input image. In C p Each channel uses independent k*k convolutions to compute the feature map, the results of the two branches are concatenated, and all features are fused using 1*1 convolutions; S4: Use a semantic segmentation network model to process the weld seam image under test and stitch it back to the original image size.

2. The method for detecting industrial weld defects based on semantic segmentation according to claim 1, characterized in that, The method for allocating the excess pixels includes: B1: Determine the basic width of the image and identify the number of redundant pixels. The basic width is calculated by integer division to determine the width value to be allocated to each segment. The number of redundant pixels is determined by modulo operation. B2: Traverse each segment of the image and determine the relationship between the index of the current segment and the number of redundant pixels; B3: Determine whether to increase the width of an image by one pixel based on the index relationship.

3. The method for detecting industrial weld defects based on semantic segmentation according to claim 1, characterized in that, The loss function is a combination of specific weights used to guide the model in accurately learning the boundaries of each category, addressing the class imbalance problem, and enhancing the ability to detect defects in small targets. The formula is as follows: ； in, , , These are the weight coefficients for cross-entropy loss (CE Loss), Dice loss (Dice Loss), and focal loss (Focal Loss), respectively.

4. The method for detecting industrial weld defects based on semantic segmentation according to claim 1, characterized in that, S1 further includes: S11: Perform an expansion operation on the rectangular region, with the expansion size set to 21*21 pixels. Calculate the boundary coordinates of the minimum bounding rectangle of the expanded region. The boundary coordinates include the row positions of the upper and lower boundaries of the rectangle. S12: Store the boundary coordinate information into the corresponding variable, generate a new rectangular region based on the calculated boundary coordinates of the minimum bounding rectangle, crop the processed original image based on the generated rectangular region, and save the cropped image.

5. The method for detecting industrial weld defects based on semantic segmentation according to claim 1, characterized in that, The original image processing in S1 also includes: connecting the binarized foreground regions, merging adjacent regions into a whole, and storing the merged region into a variable.

6. The method for detecting industrial weld defects based on semantic segmentation according to claim 1, characterized in that, The S2 further includes: for each segment, determining the start and end positions of the cropping region based on its allocated width and the current left boundary; after cropping a segment, updating the current left boundary to the right boundary of the previous cropping region.

7. The method for detecting industrial weld defects based on semantic segmentation according to claim 1, characterized in that, The processing of the dataset in S2 also includes: enhancing the dataset by random flipping and random rotation, and adding the enhanced data to the dataset.

8. The method for detecting industrial weld defects based on semantic segmentation according to claim 1, characterized in that, S4 further includes: S41: Input the image of the weld to be tested, use the semantic segmentation network model to cut the image, and record the size data of each original image of the weld to be tested and the information of the same image; S42: Perform contrast stretching, adaptive histogram equalization, and gamma correction sequentially on the cut weld seam image to enhance the image; S43: Input the enhanced image from S42 into the semantic segmentation network model and output the prediction result; S44: Based on the image number of the predicted cut, restore each slice to the position corresponding to the original weld seam image to be tested, and seamlessly restore the image to its original size.

Images