Industrial weld defect detection method based on semantic segmentation

Abstract

The invention discloses an industrial weld defect detection method based on semantic segmentation. The method comprises the following steps: S1, obtaining an original weld image for training and related parameter information, carrying out binarization processing on the original image, storing a foreground region of the processed image, selecting a region with the maximum area to carry out rectangular shape conversion, and cutting and storing a rectangular region; s2, marking the cut rectangular region image, generating a weld defect image data set, cutting the image in the weld defect image data set by using an intelligent cutting method, and distributing redundant pixels generated by cutting; s3, constructing a semantic segmentation network model according to the data set generated in S2, focusing a defect region by using an attention mechanism, performing selective convolution calculation on the defect region, and training the semantic segmentation network model by using a loss function as an index; and S4, processing the weld seam image to be detected by using the semantic segmentation network model, and splicing and restoring the weld seam image to an original image size. According to the invention, the detection precision is improved, and image processing is not limited.

Description

Technical Field

[0001] The present invention relates to the technical field of welding detection, and in particular to an industrial weld defect detection method based on semantic segmentation. Background Art

[0002] Industrial welding technology is widely used in many key fields such as aviation, aerospace, atomic energy, chemical industry, shipbuilding, electronic technology, construction, transportation, electricity, machinery manufacturing, etc., and is an indispensable part of modern industry. However, affected by various factors, defects that do not meet the design or standard requirements often appear in welds or welded joints, which are specifically manifested as imperfect, discontinuous or harmful states. These defects not only weaken the strength and stability of the welded structure, but also may pose a potential threat to the performance and safety of the welded parts. At present, although there are a variety of welding quality detection technologies, they are often limited by detection accuracy, efficiency or cost. Therefore, accurate and efficient detection of welding quality is still an indispensable and urgent part of ensuring industrial safety.

[0003] At present, in the detection of industrial X-ray weld defects, the traditional method relies on manual visual inspection, which has limited accuracy, especially the weak ability to identify small defects (such as small cracks and pores), which directly affects the safety and reliability of welded structures. With the rapid development of modern industrial production, the number of welds has increased dramatically. Traditional detection methods are time-consuming and inefficient, and it is difficult to meet the detection needs of large-scale production. Manual detection not only requires a large amount of human resources, but also requires high training costs, and the stability and consistency of the test results are difficult to guarantee, further increasing the overall cost. Manual detection is easily affected by factors such as the experience, fatigue, and concentration of the inspectors, resulting in a high rate of misjudgment and missed detection, posing a potential threat to product quality and safety. Summary of the invention

[0004] In order to solve the problems of low detection accuracy and limited image processing in the prior art of weld defect detection, the present invention proposes an industrial weld defect detection method based on semantic segmentation.

[0005] The specific technical solutions are as follows:

[0006] Industrial weld defect detection method based on semantic segmentation, including:

[0007] S1: Obtain the original weld image and related parameter information for training, perform binarization on the original image, store the foreground area of ​​the processed image, select the area with the largest area for rectangular shape conversion, crop the rectangular area and save it;

[0008] S2: annotate the cropped rectangular area image, generate a weld defect image dataset, cut the images in the weld defect image dataset using an intelligent cutting method, and allocate the redundant pixels generated by the cutting;

[0009] S3: Build a semantic segmentation network model based on the data set generated by S2, use the attention mechanism to focus on the defect area, perform selective convolution calculations on the defect area, and use the loss function as an evaluation indicator to train the semantic segmentation network model;

[0010] S4: Use the semantic segmentation network model to process the weld image to be tested and stitch it back to the original image size.

[0011] Furthermore, the intelligent cutting method includes:

[0012] A1: Get the input image parameters and determine the width and height of the input image parameters. If the width is smaller than the height, rotate the input image 90 degrees so that the width is always the larger parameter.

[0013] A2: Determine the number of image divisions, obtain the average image width based on the number of divisions, determine the ratio of width to height, and if the ratio is less than 2, resize the image to 256*256. If the ratio is greater than or equal to 2, cut the original image according to the number of image divisions, use the average image width as the width of all images, save the size and number of the cut images, and resize the cut images to 256*256.

[0014] Furthermore, the method for allocating redundant pixels includes:

[0015] B1: Determine the basic width of the image and identify the number of excess pixels. The basic width is calculated by integer division to calculate the width value that should be allocated to each segmented part, and the number of excess pixels is determined by modulus operation;

[0016] B2: traverse each segmented part of the image, based on the relationship between the index of the current part and the number of redundant pixels;

[0017] B3: Determine whether to increase the width of a certain image by one pixel based on the index relationship.

[0018] Furthermore, the semantic segmentation network model includes:

[0019] Image segmentation network: It is constructed by Transformer and U-Net, suitable for small target detection, and used to extract and transfer features;

[0020] RRC module: used for lightweight convolution, further fusing the staggered layer information after decoupling between convolution layers;

[0021] CRA module: used to enhance the spatial correlation and context capture capabilities of the model when processing image semantic segmentation tasks. It divides the input feature map into windows, generates multiple weight matrices for the divided areas and performs feature projection. It also calculates the degree of correlation between different areas based on the weight matrix, and selects areas based on the degree of correlation between different areas.

[0022] Furthermore, the loss function is a loss function combined with specific weights to guide the model to accurately learn the boundaries of each category, deal with the problem of category imbalance, and enhance the detection ability of small target defects. The loss function is expressed as:

[0023] Total Loss = λ CE CE Loss+λ Dice Dice Loss + λ Focal Focal Loss

[0024] Among them, λ CE , Dice , Focal are the weight coefficients of cross entropy loss, Dice loss and focal loss respectively.

[0025] Furthermore, S1 also includes:

[0026] S11: performing an expansion operation on the rectangular area, setting the expansion size to 21*21 pixels, and calculating the boundary coordinates of the minimum circumscribed rectangle of the expansion area, where the boundary coordinates include the row positions of the upper boundary and the lower boundary of the rectangle;

[0027] S12: storing the boundary coordinate information into corresponding variables, generating a new rectangular area according to the calculated boundary coordinates of the minimum circumscribed rectangle, cropping the processed original image according to the generated rectangular area, and saving the cropped image.

[0028] Furthermore, the original image processing in S1 further includes: connecting the foreground area after the binarization processing, merging adjacent areas into a whole, and storing the merged area into a variable.

[0029] Furthermore, S2 also includes: for each segmented part, determining the start and end positions of the cropping area according to its allocated width and the current left boundary, and after completing the cropping of a part, updating the current left boundary to the right boundary of the previous cropping area.

[0030] Furthermore, the processing of the data set in S2 also includes: enhancing the data set by random flipping and random rotation, and adding the enhanced data to the data set.

[0031] Furthermore, S4 also includes:

[0032] S41: input the weld image to be tested, use the semantic segmentation network model to cut the image, and record the size data of each original weld image to be tested and the information of the same image;

[0033] S42: performing contrast stretching, adaptive histogram equalization and gamma correction on the weld image to be tested after cutting in sequence to perform image enhancement;

[0034] S43: Input the enhanced image in S42 into the semantic segmentation network model and output the prediction result;

[0035] S44: According to the image number of the predicted cut, each slice is restored to the position corresponding to the original weld image to be measured, and the image is seamlessly restored to the original size.

[0036] The above technical solution has the following advantages or technical effects:

[0037] 1. The cutting method used in the present invention is not limited by the image size of 256. The image whose size exceeds 256 after trimming is not directly cut at least twice. Instead, the size of the image is controlled to be close to 256 after cutting, which greatly reduces the redundancy of calculation and improves the segmentation efficiency.

[0038] 2. When dealing with the situation where the width of an image cannot be divided by the expected number of cuts, the present invention ensures that the sizes of all cut images remain consistent by reasonably allocating pixel values, avoiding abrupt or uncoordinated size changes. Resizing the image afterwards will greatly reduce the possibility of distortion, making the result more accurate.

[0039] 3. In the process of constructing the semantic segmentation network model of the present invention, the detection accuracy of the present invention is significantly improved by introducing RRC and CRA technologies. Compared with the previous network model, its performance is more superior. In addition, the model adopts a semantic segmentation algorithm, which achieves detailed detection at the pixel level compared with the previous target detection method, thereby making important progress in detection accuracy and detail capture.

[0040] 4. In the construction of the loss function, the present invention introduces focal loss and performs weight distribution. The final optimized overall loss function significantly improves the detection effect of the overall model and the detection ability of small target defects. The loss function combined with specific weights is used as the evaluation indicator of the model, which can improve the model's accuracy in learning boundary recognition of different types of weld images and solve the model's ability to deal with category imbalance problems.

[0041] 5. Compared with traditional single image enhancement technologies, contrast stretching, histogram equalization and gamma correction, although each has unique advantages, it is often difficult to fully meet the needs of complex image scenes. The present invention combines contrast stretching and adaptive histogram equalization, and uses gamma correction to achieve better image enhancement effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of the method of the present invention;

[0043] Figure 2 is an image clipping flow chart of the present invention;

[0044] Figure 3 is the original uncropped image of the present invention;

[0045] Figure 4 is the cropped picture of the present invention;

[0046] Figure 5 is the image cutting flow chart of the present invention;

[0047] Figure 6 is a network structure diagram of the present invention;

[0048] Figure 7 It is a structural diagram of the Transformer Block of the present invention;

[0049] Figure 8 It is a structural diagram of the RRC module of the present invention;

[0050] Fig. 9 It is a CRA module structure diagram of the present invention;

[0051] Fig.10 is an image enhancement flow chart of the present invention;

[0052] Fig.11 It is a comparison diagram of the image effects of the present invention. DETAILED DESCRIPTION

[0053] In order to make the technical solution of the present invention clearer, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] like Figure 1 As shown, the industrial weld defect detection method based on semantic segmentation includes:

[0055] S1: Obtain the original weld image and related parameter information for training, perform binarization on the original image, store the foreground area of ​​the processed image, select the area with the largest area for rectangular shape conversion, crop the rectangular area and save it;

[0056] S2: annotate the cropped rectangular area image, generate a weld defect image dataset, cut the images in the weld defect image dataset using an intelligent cutting method, and allocate the redundant pixels generated by the cutting;

[0057] S3: Build a semantic segmentation network model based on the data set generated by S2, use the attention mechanism to focus on the defect area, perform selective convolution calculations on the defect area, and use the loss function as an evaluation indicator to train the semantic segmentation network model;

[0058] S4: Use the semantic segmentation network model to process the weld image to be tested and stitch it back to the original image size.

[0059] The present invention combines traditional visual technology with artificial intelligence deep learning methods, and proposes an image processing and defect recognition method, which can automatically and efficiently process X-ray images, and accurately identify various welding defects including copper / tungsten inclusions, cracks, lack of fusion, lack of penetration, and slag inclusions / porosity through algorithms. The implementation of the present invention will effectively overcome the difficulties brought about by manual evaluation, reduce the detection cost of the welding industry, and improve the detection speed and accuracy. It is suitable for a variety of welding types and materials, and can work stably and reliably in various industrial environments, providing strong technical support for improving welding quality and ensuring industrial safety.

[0060] The original weld image contains image numbers and some manual annotation information, which is redundant information for weld defect detection tasks and needs to be removed through preprocessing. Figure 2 As shown, the present invention provides a process for cutting weld images.

[0061] First, read the image file in the specified path and obtain the image size information, including the width and height of the image. This size information will be used in subsequent image processing operations to ensure that the processed area conforms to the actual image size.

[0062] Next, the image is binarized, and the part with pixel values ​​between 60 and 255 is identified as the foreground area, and the processed results are stored. Through this step, the foreground area is separated from the background, which is convenient for subsequent processing.

[0063] Then, connect these binary foreground regions, merge the adjacent regions into a whole, and store these merged regions into variables. This step ensures that all connected regions are correctly identified and processed.

[0064] From all connected regions, the largest region is selected as the most important foreground region and the result is stored. This ensures that the processing focus area is maximized.

[0065] Perform a rectangular shape conversion on the largest selected area to convert it into a rectangular shape. This step helps define a clear processing area for further operation.

[0066] The rectangular region is dilated to expand its boundaries, and the dilation size is set to 21x21 pixels. This step includes details that may have been missed and provides sufficient coverage for subsequent image cropping.

[0067] Calculate the boundary coordinates of the minimum bounding rectangle of the dilated area, including the row positions of the upper and lower boundaries of the rectangle. These boundary coordinate information are stored in the corresponding variables to provide the basis for generating the clipping area.

[0068] According to the calculated boundary coordinates of the minimum circumscribed rectangle, a new rectangular area is generated to define the cropped area, and the original image is cropped using the generated rectangular area. Finally, the cropped image is saved. By cropping the image, the redundant information on the original weld image is removed, and the before and after comparison is clear. Figure 3 , Figure 4 shown.

[0069] The LabelMe label making tool is used to make the corresponding pixel-level defect image annotations as a weld defect image dataset. The various defect types in the dataset include cracks, pores, slag inclusions, incomplete penetration, and incomplete melting. In practical applications, images captured from industrial field environments often show non-standardized size characteristics, and the image width is often significantly larger than its height. For example, the common size range covers 4209x1418 pixels to 3920x648 pixels. For such input images, if their resolution is significantly higher than the preset processing standard, the traditional size scaling method will inevitably lead to a significant loss of key information in the image, which will have an adverse impact on the subsequent image detection and analysis process, specifically manifested in a significant decrease in detection accuracy.

[0070] The present invention gives priority to the width dimension and proposes an intelligent cutting method based on the image aspect ratio (width-to-height ratio) to cut the preprocessed image. Figure 5 As shown, specifically including:

[0071] Get the input image parameters, determine the size of the width W and height H in the input image parameters, if the width W is smaller than the height H, rotate the input image 90 degrees so that the width is always the larger parameter;

[0072] Determine the number of image divisions split_pic_num according to the division rule. The number of image divisions split_pic_num is related to the ratio ar of width W and height H, ar=W / H. Divide the original image width by the number of divisions split_pic_num to obtain the average image width average_width. Determine the ratio of width W to height H. If the ratio of width W to height H is less than 2, resize the image to 256*256. If the ratio of width W to height H is greater than or equal to 2, cut the original image according to the number of image divisions into several equal parts corresponding to the number of image divisions. Use the average image width as the width of all images, save the image size and number after cutting, resize the cut image to 256*256, and re-enter the result into the model testing task.

[0073] The division rules are as follows:

[0074] 1.0<=ar<1.5---->split_pic_num=1

[0075] 1.5<=ar<2.5---->split_pic_num=2

[0076] 2.5<=ar<3.5---->split_pic_num=3

[0077] 3.5<=ar<4.5---->split_pic_num=4

[0078] 4.5<=ar<5.5---->split_pic_num=5

[0079] 5.5<=ar<6.5---->split_pic_num=6

[0080] 6.5<=ar<7.5---->split_pic_num=7

[0081] 7.5<=ar<8.5---->split_pic_num=8 . . .

[0085] 14.5<=ar<=15.5---->split_pic_num=15

[0086] The intelligent cutting algorithm in the present invention can flexibly adjust input images of different resolutions while effectively retaining key image information, thereby ensuring that the detection accuracy and overall system performance are improved while maintaining the integrity of image information. The prior art usually simply crops all images to a size of 256*256 pixels, and directly fills blanks for parts with a height of less than 256 pixels. When the height is less than 256, the number of images cut is N, and when the image height is greater than 256, the number of cuts is 2N. This is equivalent to the detection of an original image, and the time and resources consumed are doubled. The processing method of the present invention is not simply to crop all images directly to a size of 256*256 pixels. After the weld image cropping process has performed preliminary image cropping processing, the image height is already in an appropriate range. The intelligent cutting algorithm is used to pre-process with a size close to 256 pixels and then interpolate. It is not limited by the image height of 256. The final number of cuts is N. After cutting, the image size is controlled to near 256. Resizing the image will greatly reduce the possibility of distortion, making the result more accurate. The processing result is more natural and effective than the prior art.

[0087] The present invention proposes an intelligent cutting method based on the image aspect ratio (width-to-height ratio). The method can automatically adapt to images with different aspect ratios. By accurately calculating and dynamically adjusting the cutting strategy, it ensures that the image is effectively adjusted to a size that meets the network input requirements while maintaining the integrity of the image's key information, thereby improving the overall processing efficiency and detection accuracy.

[0088] After intelligent cutting using the division rule, in order to avoid the problem that the image pixels cannot be divided evenly, the present invention also proposes a special pixel allocation mechanism and cropping mechanism. This method is intended to deal with the situation where the image width cannot be divided evenly by the expected number of cuts. By reasonably allocating pixel values, it ensures that the sizes of all cut images remain consistent, avoiding abrupt or uncoordinated size changes.

[0089] Among them, the redundant pixel allocation mechanism includes:

[0090] Determine the basic width of the image (part_width) and identify the number of extra pixels (extra_pixels). The basic width is calculated by integer division (width / / split_parts) to calculate the basic width (part_width) that should be allocated to each segmented part to ensure that the width of each part remains consistent when ignoring extra pixels. The number of extra pixels is determined by modulus operation (width%split_parts) due to the inability to evenly distribute the extra pixels (extra_pixels). These pixels will be distributed according to specific rules in subsequent steps;

[0091] Traverse each segmented part of the image, based on the relationship between the index of the current part and the number of excess pixels;

[0092] Decide whether to increase the width of a certain image by one pixel based on the index relationship. If the current partial index is less than the number of extra pixels (extra_pixels), increase the width of the part by 1 pixel to ensure that the extra pixels are evenly distributed to the first extra_pixels parts in order.

[0093] Among them, the clipping mechanism includes:

[0094] Crop region calculation: For each segment, determine the start and end positions of the crop region based on its assigned width (including a possible additional pixel) and the current left edge (current_left). The crop region starts at current_left and ends at current_left + part_width_with_extra (including the additional pixel, if applicable).

[0095] Boundary update: After completing the cropping of a part, update the current left boundary (current_left) to the right boundary of the previous cropping area (i.e. current_left + part_width_with_extra) to ensure that the cropping of subsequent parts starts from the correct position.

[0096] For ease of understanding, the present invention provides the following examples:

[0097] If the image width cannot be divided evenly (a small number of extra pixels), set the image width to 605 pixels and require it to be divided into 3 parts.

[0098] Basic width calculation: 605 / / 3=201, the basic width of each part is 201 pixels.

[0099] Redundant pixel calculation: 605%3=2, there are 2 redundant pixels.

[0100] Excess pixel allocation: add 1 pixel to the first two parts.

[0101] Part 1: 201+1=202 pixels;

[0102] Part 2: 201+1=202 pixels;

[0103] Part 3: 201 pixels;

[0104] Segmentation result: The three parts are 202, 202, and 201 pixels respectively, and the last part has one pixel less.

[0105] Conclusion: When the image width cannot be divided evenly, the mechanism distributes the excess pixels evenly to the first few parts through intelligent allocation to avoid large uneven distribution.

[0106] Through the above implementation process, the problem of redundant pixels caused by the inability to evenly distribute pixels can be effectively solved, and the rationality of the size of each segmented part is ensured through an intelligent allocation strategy. Finally, by adjusting the aspect ratio of the image to a ratio close to 1, the image data is optimized to make it more suitable for the needs of subsequent model training, thereby improving the effect and efficiency of model processing.

[0107] After completing the above process, the dataset is enhanced by random flipping and random rotation, and the enhanced data is added to the dataset. This completes the creation of the dataset, and then the created dataset is sent to the following network.

[0108] Most existing weld defect detection schemes focus on target-level detection of defects, using target detection algorithms, which can achieve detection, but the accuracy is not as high as that of semantic segmentation. In addition, in practical applications, the size of the defect is crucial, so the detection method based on semantic segmentation proposed in the present invention can directly calibrate the specific size of the defect for the high-precision results obtained after detection, which is convenient for subsequent defect level determination.

[0109] like Figure 6 As shown, the present invention proposes a semantic segmentation neural network model RC-WeldUnet based on Resnet50 and Transformer block, which uses the image segmentation network of Transformer and U-Net architecture to adapt to small target detection. This structural design aims to make full use of the advantages of Transformer in processing global dependencies and combine U-Net's ability to process local features and efficient upsampling. The skip connections directly pass the high-resolution features of the encoder to the decoder, which helps to improve the segmentation accuracy. In the overall structure, CNN is used in the encoder part to achieve fast and efficient spatial feature extraction, and then the feature map is passed to the Transformer layer for further global information integration. In addition, the RRC module is used to further reduce the number of parameters and reduce redundant calculations. Choose Region Attention is used before TransformerBlock. This attention mechanism allows the model to focus on the target defect area when processing the image. The Transformer Block structure is as shown in Figure 7 In this way, the model can more accurately identify and segment the key parts of the image.

[0110] Some of the parameters involved in the model are shown in the following table:

[0111] Table 1: Network parameter table

[0112]

[0113]

[0114] The semantic segmentation network model RC-WeldUnet is constructed. The RRC and CRA technologies are introduced during the construction process, which significantly improves the detection accuracy of the present invention (with mIoU as an indicator), which is described in detail below.

[0115] In the weld defect detection task, the self-attention mechanism is used, which brings a lot of computational cost while obtaining a global perception. In order to improve the performance of the model, a method combining self-attention and convolution is used. However, since traditional convolution will greatly increase the number of memory accesses, the present invention considers using a lightweight convolution method. Although PConv can reduce some Flops, the present invention still believes that there is still room for reducing redundancy. For this purpose, a lightweight RRConv module (RRC module) is proposed, which is called Reduced Undancy Convolution.

[0116] The RRConv feature extraction module is a new convolution module in the entire network, which is mainly used to further fuse the staggered layer information after decoupling between layers. Figure 8 ) shows how to further optimize the cost for the redundancy of feature maps. Because feature maps between different channels have a high degree of similarity, the present invention retains part of the feature map information and only performs convolution operations on part of the feature maps.

[0117] The input data of the RRC module is I RRC ∈R H*W*C After inputting the data, the feature map is divided into two branches, and the split ratio r in this paper is a The setting changes from 1.0 at the beginning to 0.25 at the end. a , divide channel C into p and 1-C p . Channel 1-C p ∈R H*W*(1-Cp) It is the identity mapping branch, which does not perform any operations on the data and retains the original image information to alleviate the gradient vanishing problem in deep networks and promote better information flow and model training. p ∈R H*W*Cp Extract the branch for the feature, and then use C p k*k convolution kernels are used to extract features. The memory access volume is: H*W*Cp 2 *K 2 , which can avoid redundant calculations due to similar feature maps and achieve a good balance between accuracy and compression.

[0118] The current Flops is still too high and has not met expectations. Inspired by DWConv, this paper proposes a method to convert the previous C p The channels are regarded as the representatives of the entire feature map and the calculation becomes p Each channel on the feature map uses independent k*k convolutions to calculate the feature map. This method further reduces Flops to: H*W*K 2 *C p , Flops reduces C p times.

[0119] After completing the above operations, the results of the two branches are spliced ​​together, thereby enhancing the information flow between different layers and better preserving and extracting the original information. Finally, all features are fused using 1*1 convolution to obtain O∈R H *W*C The final Flops becomes: H*W*K 2 *C p +H*W*C 2 .

[0120] The present invention deals with small target defect detection tasks, so most of the content is irrelevant elements such as background. In view of this situation, the present invention designs a region selection routing mechanism CRA module (ChooseRegionAttention), which is a sparse attention mechanism, such as Fig. 9 As shown, the spatial correlation and context capture capabilities of the model are enhanced when processing image semantic segmentation tasks, thereby achieving the goal of efficient and accurate image segmentation.

[0121] For the input feature map featuremap, F∈R H*W*C , after the Transformer window partition, it will be divided into N different non-overlapping sub-regions, each of which represents an independent processing unit. For each small block, a query (Q), key (K) and value (V) are generated. This is done by combining each small block of the original feature map F with three different weight matrices W Q , W K , W V These weight matrices are learnable parameters that define a linear transformation for each type of projection.

[0122] Q=F'W q

[0123] K=F'W K

[0124] V=F'W V

[0125] Where F' is a rearranged feature block that needs to be adjusted in some dimensions to fit the weight matrix. q W K W V ∈R C*C , QKV∈R N2*HW / N2*C .

[0126] Based on the above, we then calculate the average feature of each region to generate the region-level query Q i and K j ∈R N2 *C .

[0127] Q i =avg(Q),K j =avg(k), where i and j represent different areas.

[0128] Use the generated Q i and K j In order to obtain the degree of relevance between different regions, the present invention uses Q i and K j Through matrix multiplication operation, we get D ij ∈R N2*N2 , the formula is as follows:

[0129] D ij =Q i ·(K j )T, where Q i is the query vector for region i. K j is the key vector for region j.

[0130] The adjacency matrix describes the strength of the relationship between each region and all other regions. Through this representation, each region not only depends on its internal features, but also takes into account the correlation with other regions.

[0131] Finally, all D ij The value is stored in the connection matrix D∈R N*N*N2*N2 In which, the first two dimensions N*N represent all possible ij combinations, that is, they represent a location area index. The next two dimensions N 2 *N 2 Indicates that each D ij But this tensor is not intuitive, so we reconstruct it. The dimension of the thinking tensor D is N*N*N 2*N 2 , according to the content information of D, the dimension can be changed to (N*N)*(N 2 *N 2 ), which facilitates the subsequent region selection strategy of the present invention.

[0132] The most relevant region is selected by the degree of relevance. Each element of D is obtained by the query vector Q. i and the key vector K j It is obtained by multiplying the points of N and N, which reflects the similarity or matching degree between regions in the feature space. 2 D ij , all D ij Each row of represents the interaction strength of that particular region with all other regions in the image.

[0133] The present invention further adopts the top-n strategy to refine the main areas that should be paid attention to in each area. ij The largest n values ​​in each row can be used to construct a region index matrix R i , each row in the matrix contains the indices of other regions that the corresponding region should prioritize. This step is achieved by the following operations:

[0134] R i =N(D[i,j])

[0135] Where i,j∈R N*N , the role of the N function is to select the index of the largest n elements from the adjacency matrix D[i,j], ensuring that the attention mechanism can focus on the most relevant regional features.

[0136] In addition, the model uses a semantic segmentation algorithm, specifically a loss function, which achieves detailed pixel-level detection compared to previous object detection methods, thus making important progress in detection accuracy and detail capture.

[0137] In the task of semantic segmentation of weld defects, traditional single loss functions are often unable to fully cope with all challenges. Although cross entropy loss can effectively guide the model to learn category boundaries, it is powerless when dealing with category imbalance problems; although Dice loss can enhance the detection ability of small objects, it lacks accurate learning of other category boundaries; focal loss focuses on solving the problems of category imbalance and difficult-to-classify samples, but when used alone, it may not fully utilize the global learning ability of the model.

[0138] To this end, the present invention proposes a combined loss function mechanism, which combines the above three loss functions according to specific weights as an indicator for evaluating the model, which is used to guide the update of model parameters and work together in the model training process. This combined loss function can not only guide the model to accurately learn the boundaries of each category, but also effectively deal with the problem of category imbalance, and especially enhance the detection ability of small target defects.

[0139] The specific formula is as follows:

[0140] Total Loss = λ CE CE Loss+λ Dice Dice Loss + λ Focal Focal Loss

[0141] Among them, λ CE , Dice , Focal are the weight coefficients of cross entropy loss, Dice loss and focal loss respectively.

[0142] Cross-Entropy Loss is a classic choice in classification tasks. Its core advantage is that it can effectively measure the difference between the model's predicted probability distribution and the true label distribution. In this model, for the classification problem of six categories (including background), cross-entropy loss not only ensures that the model accurately learns the boundaries of each category, but also supports an efficient gradient descent optimization process through its good differentiability.

[0143] The specific formula is:

[0144]

[0145] Where N is the total number of samples, C is the total number of categories (including background), and y nc is the true label (0 or 1), p nc Predict probabilities for the model.

[0146] Since the weld defect area is smaller than the background area and the class imbalance problem is significant, the present invention introduces the Dice coefficient loss (Dice Loss) to enhance the model's ability to detect small target defects. Dice loss directly optimizes the overlap between the predicted area and the real defect area, which is particularly effective in dealing with the class imbalance problem in semantic segmentation.

[0147] The formula is:

[0148]

[0149] Among them, Y xy is the true label binary image, is the probability map predicted by the model (after sigmoid transformation), and ò is a smoothing term to prevent the denominator from being zero.

[0150] In order to solve the problem of excessive focus on easy-to-classify samples caused by category imbalance, the present invention further integrates focal loss. This loss function dynamically adjusts the loss weights of samples of different difficulty levels, making the model more focused on difficult-to-classify samples, thereby improving the generalization ability and robustness of the overall model.

[0151] The specific form of focal loss is:

[0152]

[0153] Among them, α t Used to balance positive and negative samples, pt is the model's predicted probability of the target category, and γ is the focusing parameter for adjusting the weight of easy-to-classify samples.

[0154] For different combinations of loss functions, the present invention compares the detection accuracy (with mIoU as the indicator):

[0155] Table 2: Loss function effect comparison table

[0156]

[0157] Compared with other semantic segmentation models, the detection accuracy of the RC-WeldUnet model used in this invention (with mIoU as the indicator) has obvious advantages:

[0158] Table 3: Comparison table of semantic segmentation model experimental results

[0159]

[0160]

[0161] At this point, the entire model training phase is completed. Next, the trained model is used for weld detection. In the weld detection phase, image cutting processing must be performed for each input image. This processing is performed based on the intelligent cutting method. When executing this method, the size data of each original image and the information of the same image must be additionally recorded. This record ensures that the cut image fragments from different sources are clearly distinguished and identified, so that in subsequent image fusion, the cut fragments can be accurately reassembled into the original image based on this recorded information, thereby supporting a comprehensive analysis and evaluation of the weld quality. The specific steps are as follows:

[0162] Input the weld image to be tested, use the semantic segmentation network model to cut the image, and record the size data of each original weld image to be tested and the information of the same image.

[0163] The weld image to be tested after cutting is subjected to contrast stretching, adaptive histogram equalization and gamma correction in sequence to perform image enhancement, such as Fig.10 As shown, specifically including:

[0164] Apply contrast stretching to the image. This step enhances details by adjusting the brightness and contrast of the image so that low-brightness and high-brightness areas of the image are better displayed. The specific operation involves calculating the low and high percentile values ​​of the image brightness (for example, 2% and 98%), and then remapping the image pixel values ​​based on these values ​​to expand the brightness range and increase the contrast. For example, you can use OpenCV's cv2.convertScaleAbs() function to ensure that the type of image data is suitable for contrast stretching.

[0165] Apply adaptive histogram equalization (CLAHE) to the contrast stretched image. CLAHE enhances image contrast by performing local equalization, especially in areas of low contrast. This process works by converting the image from BGR color space to YUV color space, applying the CLAHE algorithm to the Y channel (luminance channel), and finally converting the image back to BGR color space. CLAHE parameters (such as clipLimit = 1.0 and tileGridSize = (8,8)) control the strength of the equalization and the size of the local processing, thereby improving the local contrast of the image while avoiding over-equalization.

[0166] Finally, the image is gamma corrected. Gamma correction is used to adjust the brightness of the image to optimize the visual effect of the image or adapt it to specific display needs. The image brightness is adjusted by calculating the inverse of the gamma value (1.0 / gamma), creating a lookup table (LUT), and applying it to the image data. Gamma correction helps to correct for uneven brightness that may have been introduced in the previous processing steps and ensures that the image brightness distribution is suitable for actual use.

[0167] The above-mentioned image enhancement methods, contrast stretching and adaptive histogram equalization, have poor effects alone. Combining these enhancement methods with gamma correction can achieve better results. The above-mentioned image enhancement methods can improve the test results, such as Fig.11 The following are the image comparison effects, where A is the original image; B is the effect of using histogram equalization alone; C is the effect of using contrast enhancement alone; and D is the image enhancement effect according to the above process.

[0168] The enhanced image is input into the semantic segmentation network model RC-WeldUnet, and the output prediction results are as follows:

[0169] Table 4: Model testing results

[0170] Class Background track nofu nope cir tun IoU 0.9984 0.8222 0.8125 0.8024 0.8662 0.9776 Precision 0.9977 0.8480 0.9325 0.8625 0.8649 0.9200 Recall 0.9979 0.8264 0.9319 0.8631 0.8432 0.9449 F1-score 0.9978 0.8371 0.9322 0.8628 0.8539 0.9323

[0171] Through the prediction of the above deep learning model, it is finally necessary to stitch and restore the images cut after prediction, adjust them from the unified 256x256 size during model processing back to the actual size of the original image, and ensure that the parts from the same image can be seamlessly integrated. According to the image number cut after prediction, restore each slice to the corresponding position of the original weld image to be tested, and restore the image seamlessly to its original size. The situations that need to be processed for image fusion are as follows:

[0172] Table 5: Image fusion corresponding processing

[0173]

[0174] Through image fusion technology, we can restore the multiple 256x256 pixel slices generated by the model to the size of the original image using bicubic interpolation according to the original number and size information of each slice, and fuse them according to the number to ensure seamless reconstruction of the entire image. In this process, we pay special attention to the reasonable processing of overlapping areas between slices to ensure the consistency and accuracy of the classification results, and effectively avoid misclassification that may be caused by improper splicing.

[0175] After completing the entire weld inspection process, the inspection results we obtained are not only consistent with the original image size, but also provide defect size information specific to the pixel level due to the use of semantic segmentation algorithm, further improving the accuracy and practicality of the inspection.

[0176] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. An industrial weld defect detection method based on semantic segmentation, characterized in that: include: S1: Obtain the original weld image and related parameter information for training, perform binarization on the original image, store the foreground area of ​​the processed image, select the area with the largest area for rectangular shape conversion, crop the rectangular area and save it; S2: annotate the cropped rectangular area image, generate a weld defect image dataset, cut the images in the weld defect image dataset using an intelligent cutting method, and allocate the redundant pixels generated by the cutting; S3: Build a semantic segmentation network model based on the data set generated by S2, use the attention mechanism to focus on the defect area, perform selective convolution calculations on the defect area, and use the loss function as an evaluation indicator to train the semantic segmentation network model; S4: Use the semantic segmentation network model to process the weld image to be tested and stitch it back to the original image size.

2. The method for industrial weld defect detection based on semantic segmentation according to claim 1, characterized in that: The intelligent cutting method comprises: A1: Get the input image parameters and determine the width and height of the input image parameters. If the width is smaller than the height, rotate the input image 90 degrees so that the width is always the larger parameter. A2: Determine the number of image divisions, obtain the average image width based on the number of divisions, determine the ratio of width to height, and if the ratio is less than 2, resize the image to 256*256. If the ratio is greater than or equal to 2, cut the original image according to the number of image divisions, use the average image width as the width of all images, save the size and number of the cut images, and resize the cut images to 256*256.

3. The method for industrial weld defect detection based on semantic segmentation according to claim 1, characterized in that: The method for allocating the redundant pixels comprises: B1: Determine the basic width of the image and identify the number of redundant pixels, wherein the basic width is calculated by integer division to calculate the width value to be allocated to each segmented part, and the number of redundant pixels is determined by modulo operation; B2: traverse each segmented part of the image, based on the relationship between the index of the current part and the number of redundant pixels; B3: Determine whether to increase the width of a certain image by one pixel based on the index relationship.

4. The method for industrial weld defect detection based on semantic segmentation according to claim 1, characterized in that: The semantic segmentation network model includes: Image segmentation network: It is constructed by Transformer and U-Net, suitable for small target detection, and used to extract and transfer features; RRC module: used for lightweight convolution, further fusing the staggered layer information after decoupling between convolution layers; CRA module: used to enhance the spatial correlation and context capture ability of the model when processing image semantic segmentation tasks, divide the input feature map into windows, generate multiple weight matrices for the divided areas and perform feature projection, and calculate the degree of correlation between different areas based on the weight matrix, and select areas based on the degree of correlation between different areas.

5. The method for industrial weld defect detection based on semantic segmentation according to claim 1, characterized in that: The loss function is a loss function combined with specific weights, which is used to guide the model to accurately learn the boundaries of each category, deal with the problem of category imbalance, and enhance the detection ability of small target defects. The formula is expressed as: Total Loss=λ CE ·CE Loss+λ Dice ·Dice Loss+λ Focal ·Focal Loss Among them, λ CE , Dice , Focal are the weight coefficients of cross entropy loss, Dice loss and focal loss respectively.

6. The method for industrial weld defect detection based on semantic segmentation according to claim 1, characterized in that: The S1 further comprises: S11: performing an expansion operation on the rectangular area, setting the expansion size to 21*21 pixels, and calculating the boundary coordinates of the minimum circumscribed rectangle of the expansion area, wherein the boundary coordinates include the row positions of the upper boundary and the lower boundary of the rectangle; S12: storing the boundary coordinate information into corresponding variables, generating a new rectangular area according to the calculated boundary coordinates of the minimum circumscribed rectangle, cropping the processed original image according to the generated rectangular area, and saving the cropped image.

7. The method for industrial weld defect detection based on semantic segmentation according to claim 1, characterized in that: The original image processing in S1 further includes: connecting the foreground areas after the binarization processing, merging adjacent areas into a whole, and storing the merged area into a variable.

8. The method for industrial weld defect detection based on semantic segmentation according to claim 1, characterized in that: The step S2 further includes: for each segmented part, determining the start and end positions of the cropping region according to its allocated width and the current left boundary, and after completing the cropping of a part, updating the current left boundary to the right boundary of the previous cropping region.

9. The method for industrial weld defect detection based on semantic segmentation according to claim 1, characterized in that: The processing of the data set in S2 further includes: enhancing the data set by random flipping and random rotation, and adding the enhanced data to the data set.

10. The method for industrial weld defect detection based on semantic segmentation according to claim 1, characterized in that: The S4 further comprises: S41: input the weld image to be tested, use the semantic segmentation network model to cut the image, and record the size data of each original weld image to be tested and the information of the same image; S42: performing contrast stretching, adaptive histogram equalization and gamma correction on the weld image to be tested after cutting in sequence to perform image enhancement; S43: Input the enhanced image in S42 into the semantic segmentation network model and output the prediction result; S44: According to the image number of the predicted cut, each slice is restored to the position corresponding to the original weld image to be measured, and the image is seamlessly restored to the original size.

Images