A semantic segmentation method for weld defects based on diffusion model data augmentation and improved YOLOv8-seg

Abstract

This invention discloses a semantic segmentation method for weld defects based on diffusion model data augmentation and improved YOLOv8-seg. The segmentation method includes: acquiring real sample images in industrial welding scenarios; performing lightweight fine-tuning on a pre-trained latent diffusion model to obtain a second latent diffusion model; acquiring fake sample images based on the second latent diffusion model; mixing the real sample images and fake sample images to obtain a mixed sample training set; inputting the weld image to be detected into the defect segmentation model; and outputting the defect category and pixel-level defect region of the weld defect. This invention can reduce the insufficient response to defect details when the segmentation prediction head directly predicts based on the original feature map, and can also reduce the problem of pore edge regions being missed or background regions being misclassified, making the output pixel-level defect region closer to the actual defect region in the weld image.

Description

Technical Field

[0001] This invention relates to the field of semantic segmentation of weld defects, specifically to a semantic segmentation method for weld defects based on diffusion model data augmentation and improved YOLOv8-seg. Background Technology

[0002] Semantic segmentation of weld defects typically requires identifying defects such as cracks, porosity, slag inclusions, and lack of fusion in weld images, and further outputting the pixel-level location of the defect region. Existing methods utilize the YOLOv8-seg segmentation network, which extracts multi-scale visual features from the weld image through a backbone feature extraction network. A neck feature fusion network then fuses these feature maps from different scales, and finally, the fused feature map is input into a segmentation prediction head to output the defect category and pixel-level defect region. This type of network offers good detection speed and ease of deployment, and is therefore frequently used in industrial weld defect detection scenarios.

[0003] However, in weld images, defects such as cracks and micropores are typically small in size, have weak edges, and are easily affected by weld background texture, metallic reflection, oil contamination, or noise. Existing YOLOv8-seg segmentation networks usually directly input the feature map output by the neck feature fusion network into the segmentation prediction head. Although the feature map received by the segmentation prediction head contains multi-scale fusion information, the spatial location features related to defect edge contours and minute details may have been weakened by background texture or interference regions. If the feature responses of different channels and spatial locations are not further recalibrated before the feature map enters the segmentation prediction head, it can easily lead to discontinuous segmentation of thin cracks, incomplete boundaries of micropores, and inaccurate localization of pixel-level defect regions. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention uses a coordinate attention module to perform orientation and position weighting on the original feature map output by the neck feature fusion network, generating an enhanced feature map to solve the technical problems mentioned in the background.

[0005] To achieve the above objectives, the present invention provides the following technical solution: The semantic segmentation method for weld defects based on diffusion model data augmentation and improved YOLOv8-seg includes the following steps: S1. Obtain real sample images in industrial welding scenarios; S2. Based on real sample images, perform lightweight fine-tuning on the pre-trained latent diffusion model to construct a second latent diffusion model; S3. Obtain fake sample images based on the second potential diffusion model; S4. Mix the real sample images and fake sample images to obtain a mixed sample training set; The mixed sample training set contains several sample images, each of which is either a real sample image or a fake sample image. S5. Based on the mixed sample training set, supervised training is performed on the YOLOv8-seg segmentation network with the coordinate attention module to construct a defect segmentation model for outputting weld defect categories and pixel-level defect regions. S6. Obtain the image of the weld to be inspected; S7. Input the weld image to be detected into the defect segmentation model obtained through supervised training, and output the defect category and pixel-level defect area of ​​the weld defect; In some of these implementations, real sample images of industrial welding scenarios are acquired, including: S1-1. Acquire weld images in industrial welding scenarios; S1-2. Select images containing weld defects from the weld images and mark them as defective images; S1-3. Perform pixel-level semantic mask annotation on the defective images to obtain the pixel-level semantic mask corresponding to each defective image; S1-4. Pair the defective image with its corresponding pixel-level semantic mask to construct a real sample image.

[0006] In some implementations, a second latent diffusion model is constructed by lightweight fine-tuning a pre-trained latent diffusion model based on real sample images, including: S2-1, Obtain the pre-trained potential diffusion model; S2-2. Insert a low-rank adaptive fine-tuning module into the cross-attention layer of the pre-trained latent diffusion model to construct the first latent diffusion model; S2-3. Based on the real sample images, perform lightweight fine-tuning on the first potential diffusion model to generate a second potential diffusion model.

[0007] In some of these implementations, fake sample images are obtained based on a second potential diffusion model, including: S3-1. Constructing generation prompts for weld defects; wherein, the generation prompts include defect category, defect morphology, defect size, weld background texture, and imaging environment information; S3-2. Input the generated prompt words into the second potential diffusion model; S3-3. Based on the preset sampling parameters, the generated prompt words are sampled using the second latent diffusion model to perform conditional diffusion denoising and output the defect synthesis image. S3-4. Obtain the pixel-level semantic mask corresponding to the defect-synthesized image; S3-5. Pair the defective synthetic image with its pixel-level semantic mask to generate a fake sample image.

[0008] In some implementations, real sample images and fake sample images are mixed to obtain a mixed sample training set, including: S4-1. Preprocess the fake sample images and real sample images to obtain preprocessed fake sample images and real sample images; S4-2. Based on the preset mixing ratio, select a corresponding number of images from the preprocessed fake sample images and the preprocessed real sample images respectively; S4-3. Mix the selected fake sample images and real sample images to obtain a mixed sample training set.

[0009] In some implementations, a defect segmentation model is constructed to output weld defect categories and pixel-level defect regions, including: S5-1. Construct a segmentation network to be trained for weld defect segmentation; wherein, the segmentation network to be trained includes a backbone feature extraction network, a neck feature fusion network, a coordinate attention module, and a segmentation prediction head connected in sequence; S5-2. Select the current round sample image from the mixed sample training set and input it into the segmentation network to be trained; S5-3: The backbone feature extraction network and the neck feature fusion network of the segmentation network to be trained are propagated forward to output the original feature map of the sample image in the current round; S5-4. The coordinate attention module connected to the output end of the neck feature fusion network performs orientation and position weighting processing on the original feature map to generate an enhanced feature map. S5-5. The segmentation prediction head connected to the output of the coordinate attention module performs segmentation prediction on the enhanced feature map and outputs the defect category prediction result and pixel-level defect region prediction result of the current training sample image. S5-6. Calculate the current training loss based on the difference between the defect category prediction result, the pixel-level defect region prediction result and the pixel-level semantic mask corresponding to the current training sample image. S5-7. Based on the current training loss, backpropagate and update the parameters of the segmentation network to be trained, and iteratively select the next training sample image from the mixed sample training set to repeat the training until the segmentation network to be trained meets the preset convergence condition, thus obtaining the defect segmentation model.

[0010] In some of these implementations, a training segmentation network is built for weld defect segmentation, including: S5-1-1. Select the YOLOv8-seg segmentation network; S5-1-2. Extract the backbone feature extraction network, neck feature fusion network, and segmentation prediction head from the YOLOv8-seg segmentation network; S5-1-3. Connect a coordinate attention module for orientation awareness and position enhancement of the feature map between the neck feature fusion network and the segmentation prediction head to build a training segmentation network for weld defect segmentation.

[0011] In some of these implementations, enhanced feature maps are generated, including: S5-4-1. Perform global average pooling in the horizontal and vertical directions on the original feature map to obtain the horizontal feature vector and the vertical feature vector. S5-4-2. Concatenate the horizontal and vertical feature vectors and perform channel dimensionality reduction through shared convolution to obtain the intermediate feature map. S5-4-3. Divide the intermediate feature map into a horizontal intermediate feature map and a vertical intermediate feature map according to the spatial direction. S5-4-4 Perform channel recovery and activation processing on the horizontal and vertical intermediate feature maps to obtain the horizontal and vertical attention weights corresponding to the channel and spatial positions of the original feature maps. S5-4-5. Multiply the horizontal attention weights and vertical attention weights element-wise with the feature values ​​of the corresponding channels and spatial positions in the original feature map to generate an enhanced feature map.

[0012] This invention provides a semantic segmentation method for weld defects based on diffusion model data augmentation and improved YOLOv8-seg, which has the following beneficial effects: This invention integrates a coordinate attention module between the neck feature fusion network of the YOLOv8-seg segmentation network and the segmentation prediction head. This allows the original feature map output by the neck feature fusion network to undergo orientation and position weighting before being input into the segmentation prediction head, resulting in an enhanced feature map. Based on this enhanced feature map, the segmentation prediction head outputs the weld defect category and pixel-level defect region. This ensures that the feature map entering the segmentation prediction head retains more feature information related to the weld defect edge contour, the extension location of fine cracks, and the contour location of micropores, reducing the problem of insufficient response to defect details when the segmentation prediction head directly predicts based on the original feature map. Furthermore, the coordinate attention module recalibrates the feature responses of different channels and spatial locations in the original feature map by using horizontal and vertical attention weights. This ensures that spatial locations associated with weld defects such as slender cracks and micropores maintain a high response, while spatial locations associated with background textures, reflective interference, or noise receive a lower response. Therefore, when segmenting slender cracks, it reduces the breakage of defect areas along the crack's extension direction; and when segmenting micropores, it reduces the problem of missing pore edge areas or misclassifying background areas, making the output pixel-level defect areas closer to the actual defect areas in the weld image. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating the semantic segmentation method for weld defects based on diffusion model data augmentation and improved YOLOv8-seg according to the present invention. Figure 2 This is a schematic diagram of the training process of the defect segmentation model described in this invention; Figure 3 This is a schematic diagram illustrating the construction process of the segmentation network to be trained according to the present invention; Figure 4 This is a schematic diagram of the process for generating the enhanced feature map according to the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] First, the prior art and related concepts involved in the embodiments of the present invention will be described: YOLOv8-seg: An existing image segmentation network built on the YOLOv8 network framework, capable of classifying and segmenting target regions in input images. Unlike target detection networks that only output bounding boxes, YOLOv8-seg can output not only the target category but also the pixel-level mask corresponding to the target region, making it suitable for image segmentation tasks that require determining the target contour or the location of the target region.

[0016] Example 1: Please refer to Figure 1 This invention provides a semantic segmentation method for weld defects based on diffusion model data augmentation and improved YOLOv8-seg, comprising the following steps: S1. Obtain real sample images in industrial welding scenarios; S2. Based on real sample images, perform lightweight fine-tuning on the pre-trained latent diffusion model to construct a second latent diffusion model; S3. Obtain fake sample images based on the second potential diffusion model; S4. Mix the real sample images and fake sample images to obtain a mixed sample training set; The mixed sample training set contains several sample images, each of which is either a real sample image or a fake sample image. S5. Based on the mixed sample training set, supervised training is performed on the YOLOv8-seg segmentation network with the coordinate attention module to construct a defect segmentation model for outputting weld defect categories and pixel-level defect regions. S6. Obtain the image of the weld to be inspected; Specifically, the weld image to be detected can be acquired in real time by an industrial camera or from an offline weld image library.

[0017] S7. Input the weld image to be detected into the defect segmentation model obtained through supervised training, and output the defect category and pixel-level defect area of ​​the weld defect; The defect categories include at least one of cracks, porosity, slag inclusions, and lack of fusion. The pixel-level defect region represents the range of pixel locations of weld defects in the weld image to be detected. Further, based on the pixel-level defect region, at least one of the following can be extracted as the semantic segmentation result of the weld defect: a set of defect region contour points, defect area, defect length and width dimensions, and a confidence score. The defect region contour point set can be represented by a sequence of polygon vertices; the defect area can be represented by the number of pixels or converted to actual physical dimensions based on camera calibration parameters; and the confidence score characterizes the reliability of the defect segmentation model's identification results for the corresponding defect region.

[0018] In this embodiment, the pre-trained latent diffusion model is lightly fine-tuned using real sample images, and fake sample images with pixel-level semantic masks are generated based on the second latent diffusion model, so that the mixed sample training set contains both real labeled samples and extended generated samples. Then, the mixed sample training set is used to supervise the training of the YOLOv8-seg segmentation network with the coordinate attention module, so that the defect segmentation model can output the weld defect category and pixel-level defect region based on the weld image to be detected.

[0019] Example 2: See Figures 3 to 4 The technical solution of this embodiment 2 differs from that of embodiment 1 in that it discloses the specific implementation steps of each step in embodiment 1.

[0020] Specifically, in this embodiment, step S1 includes: S1-1. Acquire weld images in industrial welding scenarios; Specifically, the industrial welding scenarios include weld imaging scenarios under different welding processes, different lighting conditions, and different plate thicknesses.

[0021] S1-2. Select images containing weld defects from the weld images and mark them as defective images; The weld defects include at least one of the following: cracks, porosity, slag inclusions, and lack of fusion.

[0022] S1-3. Perform pixel-level semantic mask annotation on the defective images to obtain the pixel-level semantic mask corresponding to each defective image; The pixel-level semantic mask annotations include the boundary contours of the defect region and the background region, as well as the defect category. The defect region is annotated with a closed contour according to the actual boundary of the defect in the image and assigned a corresponding defect category. The background region is labeled as the background category. S1-4. Pair the defective image with its corresponding pixel-level semantic mask to construct a real sample image.

[0023] In this embodiment, by first screening defective images, then performing pixel-level semantic mask annotation on the defective images, and pairing the defective images with their corresponding pixel-level semantic masks, the constructed real sample images simultaneously contain weld defect image content and corresponding pixel-level annotation information, providing an annotation basis for subsequent fine-tuning of the potential diffusion model and supervised training of the defect segmentation model.

[0024] Specifically, in this embodiment, step S2 includes: S2-1, Obtain the pre-trained potential diffusion model; S2-2. Insert a low-rank adaptive fine-tuning module into the cross-attention layer of the pre-trained latent diffusion model to construct the first latent diffusion model; Specifically, the pre-trained latent diffusion model represents a latent diffusion model pre-trained on a large-scale image dataset; while the low-rank adaptive fine-tuning module is used to introduce trainable low-rank decomposition matrix parameters without updating the backbone parameters of the pre-trained latent diffusion model as a whole.

[0025] In this embodiment, the low-rank adaptive fine-tuning module can be set in the cross-attention layer of the pre-trained latent diffusion model, and its low-rank decomposition matrix is ​​expressed as: ; in, The weight increment of the original weight matrix W is represented by B and A, which represent low-rank matrices; the fine-tuned weight matrix can be expressed as: ; Where W′ represents the updated weight matrix that participates in inference after inserting the low-rank adaptive fine-tuning module.

[0026] S2-3. Based on the real sample images, the first potential diffusion model is lightly fine-tuned to generate the second potential diffusion model; Specifically, during the lightweight fine-tuning process, the backbone parameters of the pre-trained latent diffusion model are fixed, and gradient updates are performed only on the low-rank decomposition matrix parameters in the low-rank adaptive fine-tuning module.

[0027] In this embodiment, the backbone parameters of the pre-trained latent diffusion model include at least one of U-Net encoder, U-Net decoder, VAE encoder / decoder, and text encoder; the pre-trained latent diffusion model is lightly fine-tuned using real sample images to enable it to learn the texture features, morphological features, and shape distribution patterns of pixel-level semantic masks of weld defects, thereby obtaining a second latent diffusion model adapted to the weld defect image generation task.

[0028] Furthermore, the texture features include at least one of the jagged edges of cracks and the circular dark spots of pores; the morphological features include at least one of the elongated cracks and the clustered pores.

[0029] Specifically, in this embodiment, step S3 includes: S3-1. Constructing generation prompts for weld defects; wherein, the generation prompts include defect category, defect morphology, defect size, weld background texture, and imaging environment information; Specifically, the generated prompts are in the form of natural language descriptions, such as: "a close-up photograph of a weld seam surface with a longitudinal crack defect, metallic texture, industrial workshop lighting, high resolution"; or in the form of a structured tag sequence, which includes at least: [defect category], [defect length], [defect width], [background material], and [lighting conditions].

[0030] Among them, the defect category is used to define the type of weld defect to be generated, the defect length and defect width are used to define the size range of the weld defect to be generated, and the background material and lighting conditions are used to define the imaging background of the weld image to be generated.

[0031] S3-2. Input the generated prompt words into the second potential diffusion model; S3-3. Based on the preset sampling parameters, the generated prompt words are sampled using the second latent diffusion model to perform conditional diffusion denoising and output the defect synthesis image. Specifically, the preset sampling parameters include a guiding coefficient, a sampling step count, and a sampler type; wherein, the guiding coefficient is used to control the degree of matching between the defect synthesis image and the generated prompt words of the weld defect, and the sampling step count is used to control the number of iterations of the diffusion denoising process.

[0032] In this embodiment, the guiding factor is set to 6.0 to 9.0, the number of sampling steps is set to 25 to 50, and the sampler is DPM++2MKarras; as a preferred embodiment, the guiding factor is set to 7.5 and the number of sampling steps is set to 30.

[0033] S3-4. Obtain the pixel-level semantic mask corresponding to the defect-synthesized image; Specifically, the pixel-level semantic mask is obtained through a mask synchronization generation branch connected to the second latent diffusion model. The mask synchronization generation branch takes the intermediate features of the generated prompt words and the defect synthesis image as input and outputs a pixel-level semantic mask corresponding to the defect region in the defect synthesis image.

[0034] S3-5. Pair the defective synthetic image with its pixel-level semantic mask to generate a fake sample image; In this embodiment, the pixel-level semantic mask corresponds to the defect region of the defect synthesis image and is used to form a fake sample image that can participate in supervised training; the fake sample image refers to the training sample image generated by the second latent diffusion model and paired with the pixel-level semantic mask, and does not refer to an unlabeled or invalid image.

[0035] Specifically, in this embodiment, step S4 includes: S4-1. Preprocess the fake sample image and the real sample image to obtain the preprocessed fake sample image and real sample image; Specifically, the preprocessing includes normalization processing and size preprocessing; wherein, normalization processing means normalizing the image pixel values ​​to the [0,1] range, and size preprocessing means scaling the image to a preset input size.

[0036] In this embodiment, the preset input size is 640×640 pixels.

[0037] S4-2. Based on the preset mixing ratio, select a corresponding number of images from the preprocessed fake sample images and the preprocessed real sample images respectively; S4-3. Mix the selected fake sample images and real sample images to obtain a mixed sample training set; In this embodiment, the preset mixing ratio is preferably 3:1; by mixing fake sample images with real sample images, it is possible to increase the sample diversity of defect morphology and imaging conditions while preserving the distribution of real sample images.

[0038] In this embodiment, by performing the same preprocessing on both fake and real sample images and constructing a mixed sample training set according to a preset mixing ratio, the sample images input to the segmentation network to be trained have a uniform pixel scale and image size. At the same time, by introducing fake sample images to supplement the number of real sample images, the mixed sample training set can cover more weld defect morphologies and imaging conditions.

[0039] Specifically, in this embodiment, step S5 includes: S5-1. Construct a segmentation network to be trained for weld defect segmentation; wherein, the segmentation network to be trained includes a backbone feature extraction network, a neck feature fusion network, a coordinate attention module, and a segmentation prediction head connected in sequence; S5-2. Select the current round sample image from the mixed sample training set and input it into the segmentation network to be trained; Specifically, the current round sample image is either a real sample image or a fake sample image in the mixed sample training set; the current round sample image carries a corresponding pixel-level semantic mask, which is used to calculate the segmentation loss in the subsequent supervised training process.

[0040] S5-3: The backbone feature extraction network and the neck feature fusion network of the segmentation network to be trained are propagated forward to output the original feature map of the sample image in the current round; Specifically, the original feature map is denoted as: ; Where F represents the original feature map output by the neck feature fusion network; C represents the number of channels in the original feature map; H represents the height of the original feature map; and W represents the width of the original feature map.

[0041] S5-4. The coordinate attention module connected to the output end of the neck feature fusion network performs orientation and position weighting processing on the original feature map to generate an enhanced feature map. S5-5. The segmentation prediction head connected to the output of the coordinate attention module performs segmentation prediction on the enhanced feature map and outputs the defect category prediction result and pixel-level defect region prediction result of the current training sample image. A coordinate attention module is connected between the neck feature fusion network of the YOLOv8-seg segmentation network and the segmentation prediction head. This coordinate attention module performs orientation and position weighting on the original feature map output by the neck feature fusion network to generate an enhanced feature map. The segmentation prediction head then outputs the weld defect category and pixel-level defect region based on the enhanced feature map. This allows the segmentation prediction process to enhance the response of weld defect edge contours and minute detail features before entering the prediction head.

[0042] S5-6. Calculate the current training loss based on the difference between the defect category prediction result, the pixel-level defect region prediction result and the pixel-level semantic mask corresponding to the current training sample image. In this embodiment, the current training loss can be selected to consist of a defect category loss and a pixel-level defect region loss. The defect category loss measures the difference between the predicted defect category and the corresponding defect category in the pixel-level semantic mask, while the pixel-level defect region loss measures the pixel-level difference between the predicted pixel-level defect region and the defect region in the pixel-level semantic mask. Specifically, the defect category loss uses cross-entropy loss, and the pixel-level defect region loss can use binary cross-entropy loss.

[0043] S5-7. Based on the current training loss, backpropagate and update the parameters of the segmentation network to be trained, and iteratively select the next training sample image from the mixed sample training set to repeat the training until the segmentation network to be trained meets the preset convergence condition, thus obtaining the defect segmentation model.

[0044] Specifically, during supervised training, the mixed sample training set is divided into a training set and a validation set. The training set is used to train the segmentation network to be trained by forward and backward propagation, and the segmentation accuracy is evaluated on the validation set. When the improvement of the validation set mIoU is less than a preset threshold for a set of consecutive preset rounds, an early stopping mechanism is triggered, and the model weights with the highest mIoU on the validation set are rolled back.

[0045] In this embodiment, the training parameters include at least one of the following: optimizer, initial learning rate, weight decay coefficient, batch size, training epochs, learning rate decay strategy, and early stopping mechanism. The optimizer is AdamW, the initial learning rate is set to 0.001, the weight decay coefficient is set to 0.0005, the batch size is set to 8 to 16, the training epochs are set to 100 to 200, and a cosine annealing learning rate decay strategy is adopted. The validation set mIoU is used as the evaluation metric. When the mIoU improves by less than 0.1% for 10 consecutive epochs, the early stopping mechanism is triggered, and the optimal model weights are rolled back.

[0046] Step S5-1 further includes: S5-1-1. Select the YOLOv8-seg segmentation network; Specifically, the YOLOv8-seg segmentation network is an existing segmentation network used for category recognition and pixel-level region segmentation of target regions in input images, and in this embodiment it is used as the base network for weld defect segmentation; S5-1-2. Extract the backbone feature extraction network, neck feature fusion network, and segmentation prediction head from the YOLOv8-seg segmentation network; In this embodiment, the backbone feature extraction network is used to extract multi-scale visual features of the weld image, the neck feature fusion network is used to fuse feature maps of different scales, and the segmentation prediction head is used to output defect category prediction results and pixel-level defect region prediction results. The backbone feature extraction network includes at least one of Conv layer, C2f module, and SPPF spatial pyramid pooling layer; the neck feature fusion network includes a PAN-FPN structure and fuses P3, P4, and P5 multi-layer features.

[0047] S5-1-3. Connect a coordinate attention module for orientation awareness and position enhancement of the feature map between the neck feature fusion network and the segmentation prediction head to build a training segmentation network for weld defect segmentation. Specifically, the coordinate attention module is positioned between the neck feature fusion network and the segmentation prediction head, and is used to perform orientation awareness and position enhancement on the feature map output by the neck feature fusion network before it enters the segmentation prediction head.

[0048] In this embodiment, the channel dimensionality reduction scaling factor of the coordinate attention module is set to 16, and the window size of the global average pooling corresponds to the spatial size of the input feature map.

[0049] Step S5-4 further includes: S5-4-1. Perform global average pooling in the horizontal and vertical directions on the original feature map to obtain the horizontal feature vector and the vertical feature vector. Specifically, the calculation results of global average pooling along the horizontal and vertical directions can be expressed as: ; in, Represents the feature vector in the horizontal direction; Represents the eigenvectors in the vertical direction; This indicates global average pooling performed along the horizontal direction; This indicates global average pooling performed along the vertical direction.

[0050] S5-4-2. Concatenate the horizontal and vertical feature vectors and perform channel dimensionality reduction through shared convolution to obtain the intermediate feature map. Specifically, the horizontal and vertical feature vectors are concatenated along the spatial dimension and then input into a shared 1×1 convolution for channel dimensionality reduction to obtain an intermediate feature map: ; Where f represents the intermediate feature map; This indicates the concatenation of horizontal and vertical feature vectors; Represents a 1×1 convolution; This represents a non-linear activation function.

[0051] In this embodiment, the nonlinear activation function can be the SiLU activation function; the channel dimensionality reduction scaling factor of the shared convolution is 16.

[0052] S5-4-3. Divide the intermediate feature map into a horizontal intermediate feature map and a vertical intermediate feature map according to the spatial direction. Specifically, the horizontal intermediate feature map is used to generate horizontal attention weights, and the vertical intermediate feature map is used to generate vertical attention weights.

[0053] S5-4-4 Perform channel recovery and activation processing on the horizontal and vertical intermediate feature maps to obtain the horizontal and vertical attention weights corresponding to the channel and spatial positions of the original feature maps. Specifically, the horizontal attention weights and vertical attention weights are calculated as follows: ; in, Indicates the attention weight in the horizontal direction; Indicates vertical attention weights; This represents a 1×1 convolution used to generate the horizontal attention weights; This represents a 1×1 convolution used to generate vertical attention weights; This represents the Sigmoid activation function.

[0054] It should be noted that the horizontal and vertical attention weights are not fixed weights shared across the entire image. The horizontal attention weight is used to characterize the feature response weights corresponding to different channels and horizontal positions in the original feature image, while the vertical attention weight is used to characterize the feature response weights corresponding to different channels and vertical positions in the original feature image, so that different feature values ​​in the original feature image can be matched with corresponding attention weights according to their channel positions and spatial positions.

[0055] S5-4-5. Multiply the horizontal attention weights and vertical attention weights element-wise with the feature values ​​of the corresponding channels and spatial positions in the original feature map to generate an enhanced feature map. Specifically, the enhanced feature map is calculated as follows: ; Wherein, F′ represents the enhanced feature map; F represents the feature map; This indicates element-wise multiplication.

[0056] Furthermore, for any eigenvalue in the original feature map Its weighted eigenvalues ​​can be expressed as: ; in, This represents the feature value at the c-th channel, h-th row, and w-th column position in the original feature map; This represents the horizontal attention weight corresponding to the c-th channel and the h-th row position; This represents the vertical attention weight corresponding to the position in the c-th channel and the w-th column. This represents the enhanced eigenvalues ​​after element-wise multiplication.

[0057] In this embodiment, the enhanced feature map is used to enhance the expressive ability of weld defect edge contours and minute detail features, and is especially suitable for the feature preservation of small-sized defects such as slender cracks and micropores.

[0058] In this embodiment, by matching the different feature values ​​in the original feature map with the corresponding horizontal and vertical attention weights, the feature responses of different channels and different spatial locations can be recalibrated, so that the spatial locations related to the weld defect edge contour and minute details maintain a high response, while the spatial locations related to background texture, reflective interference or noise obtain a low response, thereby obtaining an enhanced feature map.

[0059] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A welding seam defect semantic segmentation method based on diffusion model data enhancement and improved YOLOv8-seg, characterized in that, include: S1. Obtain real sample images in industrial welding scenarios; S2. Based on real sample images, perform lightweight fine-tuning on the pre-trained latent diffusion model to construct a second latent diffusion model; S3. Obtain fake sample images based on the second potential diffusion model; S4. Mix the real sample images and fake sample images to obtain a mixed sample training set; The mixed sample training set contains several sample images, each of which is either a real sample image or a fake sample image. S5. Based on the mixed sample training set, supervised training is performed on the YOLOv8-seg segmentation network with the coordinate attention module to construct a defect segmentation model for outputting weld defect categories and pixel-level defect regions. S6. Obtain the image of the weld to be inspected; S7. Input the weld image to be detected into the defect segmentation model obtained through supervised training, and output the defect category and pixel-level defect area of ​​the weld defect.

2. The semantic segmentation method for weld defects based on diffusion model data augmentation and improved YOLOv8-seg according to claim 1, characterized in that, Acquire real sample images from industrial welding scenarios, including: S1-1. Acquire weld images in industrial welding scenarios; S1-2. Select images containing weld defects from the weld images and mark them as defective images; S1-3. Perform pixel-level semantic mask annotation on the defective images to obtain the pixel-level semantic mask corresponding to each defective image; S1-4. Pair the defective image with its corresponding pixel-level semantic mask to construct a real sample image.

3. The semantic segmentation method for weld defects based on diffusion model data augmentation and improved YOLOv8-seg according to claim 1, characterized in that, Based on real sample images, a lightweight fine-tuning of the pre-trained latent diffusion model is performed to construct a second latent diffusion model, including: S2-1, Obtain the pre-trained potential diffusion model; S2-2. Insert a low-rank adaptive fine-tuning module into the cross-attention layer of the pre-trained latent diffusion model to construct the first latent diffusion model; S2-3. Based on the real sample images, perform lightweight fine-tuning on the first potential diffusion model to generate a second potential diffusion model.

4. The semantic segmentation method for weld defects based on diffusion model data augmentation and improved YOLOv8-seg according to claim 1, characterized in that, Fake sample images are obtained based on the second potential diffusion model, including: S3-1. Constructing generation prompts for weld defects; wherein, the generation prompts include defect category, defect morphology, defect size, weld background texture, and imaging environment information; S3-2. Input the generated prompt words into the second potential diffusion model; S3-3. Based on the preset sampling parameters, the generated prompt words are sampled using the second latent diffusion model to perform conditional diffusion denoising and output the defect synthesis image. S3-4. Obtain the pixel-level semantic mask corresponding to the defect-synthesized image; S3-5. Pair the defective synthetic image with its pixel-level semantic mask to generate a fake sample image.

5. The semantic segmentation method for weld defects based on diffusion model data augmentation and improved YOLOv8-seg according to claim 1, characterized in that, By mixing real and fake sample images, a mixed sample training set is obtained, which includes: S4-1. Preprocess the fake sample images and real sample images to obtain preprocessed fake sample images and real sample images; S4-2. Based on the preset mixing ratio, select a corresponding number of images from the preprocessed fake sample images and the preprocessed real sample images respectively; S4-3. Mix the selected fake sample images and real sample images to obtain a mixed sample training set.

6. The semantic segmentation method for weld defects based on diffusion model data augmentation and improved YOLOv8-seg according to claim 1, characterized in that, Construct a defect segmentation model for outputting weld defect categories and pixel-level defect regions, including: S5-1. Construct a segmentation network to be trained for weld defect segmentation; wherein, the segmentation network to be trained includes a backbone feature extraction network, a neck feature fusion network, a coordinate attention module, and a segmentation prediction head connected in sequence; S5-2. Select the current round sample image from the mixed sample training set and input it into the segmentation network to be trained; S5-3: The backbone feature extraction network and the neck feature fusion network of the segmentation network to be trained are propagated forward to output the original feature map of the sample image in the current round; S5-4. The coordinate attention module connected to the output end of the neck feature fusion network performs orientation and position weighting processing on the original feature map to generate an enhanced feature map. S5-5. The segmentation prediction head connected to the output of the coordinate attention module performs segmentation prediction on the enhanced feature map and outputs the defect category prediction result and pixel-level defect region prediction result of the current training sample image. S5-6. Calculate the current training loss based on the difference between the defect category prediction result, the pixel-level defect region prediction result and the pixel-level semantic mask corresponding to the current training sample image. S5-7. Based on the current training loss, backpropagate and update the parameters of the segmentation network to be trained, and iteratively select the next training sample image from the mixed sample training set to repeat the training until the segmentation network to be trained meets the preset convergence condition, thus obtaining the defect segmentation model.

7. The semantic segmentation method for weld defects based on diffusion model data augmentation and improved YOLOv8-seg according to claim 6, characterized in that, Constructing a training network for weld defect segmentation, including: S5-1-1. Select the YOLOv8-seg segmentation network; S5-1-2. Extract the backbone feature extraction network, neck feature fusion network, and segmentation prediction head from the YOLOv8-seg segmentation network; S5-1-3. Connect a coordinate attention module for orientation awareness and position enhancement of the feature map between the neck feature fusion network and the segmentation prediction head to build a training segmentation network for weld defect segmentation.

8. The semantic segmentation method for weld defects based on diffusion model data augmentation and improved YOLOv8-seg according to claim 7, characterized in that, Generate enhanced feature maps, including: S5-4-1. Perform global average pooling in the horizontal and vertical directions on the original feature map to obtain the horizontal feature vector and the vertical feature vector. S5-4-2. Concatenate the horizontal and vertical feature vectors and perform channel dimensionality reduction through shared convolution to obtain the intermediate feature map. S5-4-3. Divide the intermediate feature map into a horizontal intermediate feature map and a vertical intermediate feature map according to the spatial direction. S5-4-4 Perform channel recovery and activation processing on the horizontal and vertical intermediate feature maps to obtain the horizontal and vertical attention weights corresponding to the channel and spatial positions of the original feature maps. S5-4-5. Multiply the horizontal attention weights and vertical attention weights element-wise with the feature values ​​of the corresponding channels and spatial positions in the original feature map to generate an enhanced feature map.

Images