Steel plate surface defect image generation method based on de-noising diffusion probability model and visual Transform
By using a method based on a denoised diffusion probability model and a visual Transformer, the problems of low quality and insufficient global dependencies in the generation of defect samples on steel plate surfaces were solved, resulting in high-quality defect samples and improving the performance and robustness of the defect detection model.
Patent Information
- Application Number
- CN202610057540.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-16
- Publication Date
- 2026-05-01
AI Technical Summary
Existing technologies suffer from poor sample quality, lack of detail, and insufficient ability to model global dependencies when generating surface defect samples of steel plates. In particular, the image quality generated is poor when reconstructing complex texture structures and in noisy environments.
A method based on a denoising diffusion probability model and a visual Transformer is adopted. The image is converted into Gaussian noise through multi-step noise addition. Global features are extracted using a visual Transformer and enhanced by a channel-self-space attention module. Temporal embedding features are used for conditional guidance to gradually denoise and generate target defect samples.
More realistic and detailed steel plate defect samples were generated, which significantly improved the generalization ability and robustness of the defect detection model, especially in maintaining high quality in complex noise environments, and enhanced the ability to focus on key areas and capture details.
Smart Images
Figure CN121962343A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of industrial surface defect detection technology, and in particular to a method for generating images of steel plate surface defects based on a denoising diffusion probability model and a visual Transformer. Background Technology
[0002] In the field of industrial inspection, advancements in manufacturing technology have led to continuous optimization of steel plate production processes, resulting in a significant decrease in the number of high-quality defect samples and making it difficult to obtain such samples. This poses a challenge to deep learning models, particularly regarding the shortage of high-quality samples. Therefore, addressing this issue has become a key research problem in this field.
[0003] With the emergence of generative models, this problem has been effectively alleviated. For example, GANs generate defect samples through adversarial training between the generator and discriminator. However, GAN-based methods often encounter mode collapse during training, resulting in generated samples lacking detail and affecting their quality. In recent years, the emergence of denoising probabilistic diffusion models has provided a new solution for steel plate surface defect samples. Unlike GANs, denoising probabilistic diffusion models (DDPM) generate samples by progressively adding noise to the data and then progressively denoising it in reverse, avoiding the mode collapse problem of GANs. At the same time, it can generate higher quality and more stable samples, demonstrating superior performance compared to GANs.
[0004] Despite the popularity of DDPM in industrial inspection, it still faces several challenges in generating samples. First, DDPM's U-Net architecture, which favors local feature modeling and lacks effective global dependency capture, leads to inconsistent texture structures in the generated images, affecting sample quality. Second, industrial images often contain strong noise that can obscure texture details, causing information loss during complex texture reconstruction in DDPM, thus impacting image quality and defect detection performance. Summary of the Invention
[0005] This invention proposes a method for generating images of steel plate surface defects based on a denoising diffusion probability model and a visual Transformer, in order to solve the problems of low quality of generated samples, lack of detail, and insufficient global dependency modeling ability in existing diffusion model-based methods for generating images of steel plate surface defects.
[0006] This invention provides a method for generating images of surface defects in steel plates based on a denoising diffusion probability model and a visual Transformer. The method includes the following steps:
[0007] Step S1: Convert the original defect image into a Gaussian noise image through multi-step noise addition, and determine its corresponding time step information;
[0008] Step S2: Extract global features of the Gaussian noise image using a visual Transformer; wherein the visual Transformer integrates a channel-self-space attention module to enhance attention on features in both channel and spatial dimensions;
[0009] Step S3: Fuse the global features with the temporal embedding features generated based on the time step information to obtain conditional guidance features;
[0010] Step S4: Input the Gaussian noise image and the conditional guided features into the denoising diffusion probability model, and perform an iterative reverse denoising process based on the conditional guided features through the denoising diffusion probability model to gradually remove noise and generate the target defect sample image.
[0011] Furthermore, the specific method for converting the original defect image into a Gaussian noise image through multi-step noise addition and determining its corresponding time step information in step S1 includes:
[0012] Original defect image In discrete time steps Transformed into a latent representation of isotropic Gaussian noise Specifically:
[0013] At each time step t, towards A small amount of Gaussian noise is injected into it, and its conditional probability distribution is defined as:
[0014]
[0015] Where N represents a normal distribution, It is the identity matrix. A pre-defined variance scheduling table; This means that the mean at the current time is the state at the previous time. Scaling by a factor , here It is a hyperparameter that determines the noise intensity at each time step;
[0016] The process involves a total of [number] steps. Each of the diffusion steps introduces independent and identically distributed Gaussian noise into the image, gradually destroying the image structure and generating a series of images with varying degrees of degradation; specifically, the first... The variance of the added noise is denoted as ,in , and with the diffusion step It gradually increases with the increase of ; the entire forward diffusion process can be formally defined as:
[0017]
[0018] in, For the original image, Indicates the first The noisy image generated in step 1 represents the first step. Noisy image generated step by step Noise that follows a standard normal distribution;
[0019] As time goes by From 0 to As the amount of original information contained in an image gradually decreases and approaches zero, the amount of information in the image gradually decreases and approaches zero. hour, The distribution will be very close to a standard isotropic Gaussian distribution. At this time, the original image The structural information was almost completely obscured by noise.
[0020] Further, in step S2, the global features of the Gaussian noise image are extracted using a visual Transformer; wherein the visual Transformer integrates a channel-self-space attention module to enhance the attention of features in both channel and spatial dimensions. Specific methods for this include:
[0021] S2-1: Input image x t Adjust to a preset size and divide into a sequence of image blocks, then obtain an image block embedding sequence through linear mapping;
[0022] S2-2: The image block embedding sequence is reshaped into a two-dimensional feature map, and then processed in series through the channel attention submodule, self attention submodule and spatial attention submodule in the channel-space self attention module;
[0023] The channel attention submodule is used to model the importance of each channel feature and generate channel attention weights;
[0024] The self-attention submodule is used to explicitly capture the long-distance dependency between any two spatial locations within the feature map, generating a context-aware feature representation;
[0025] The spatial attention submodule is used to identify and emphasize key spatial regions in the feature map and generate spatial attention weights;
[0026] S2-3: The feature map enhanced by the channel-space self-attention module is flattened back into a sequence, and positional encoding and learnable classification labels are added to form a complete input sequence;
[0027] S2-4: The input sequence is fed into a module consisting of a multi-layer Transformer encoder for processing, and the output of the classification label is used as the global feature representation of the noisy image.
[0028] Furthermore, the specific methods used by the channel attention submodule to model the importance of features among channels and generate channel attention weights include:
[0029] Given input feature map The channel attention submodule first performs global average pooling (avg) and global max pooling (max) operations in parallel along the spatial dimension. Aggregate information and generate two separate results. Channel computed descriptor:
[0030]
[0031]
[0032] Where F represents the input feature map, c represents the channel index, H and W represent the height and width of the feature map, and F(:,i,j) represents the feature value of the c-th channel at position (i,j).
[0033] The two descriptors are then fed into a multilayer perceptron (MLP) with shared parameters; this MLP contains two fully connected layers, the first for dimensionality reduction and the second for channel recovery; the MLP output features from the two pooling paths are element-wise summed, and then the final channel attention weights are generated through a sigmoid activation function. :
[0034]
[0035] in, This represents the Sigmoid activation function;
[0036] Learned channel attention weights Compared with the original input feature map Element-wise multiplication (broadcasting is used for spatial dimensions) is performed to obtain a feature map that has been weighted by channel attention. :
[0037] .
[0038] Furthermore, the self-attention submodule is used to explicitly capture long-distance dependencies between any two spatial locations within the feature map, and the specific method for generating context-aware feature representations includes:
[0039] Following the channel attention module, a self-attention mechanism is introduced to further refine the feature maps. The aim is to explicitly capture long-distance dependencies between any two spatial locations within a feature map;
[0040] Use standard scaled dot product self-attention;
[0041] First, the feature map after channel attention enhancement. The query, key, and value are generated through three independent linear transformations; these tensors are reshaped before attention computation to facilitate the calculation of the dot product attention; specifically as follows:
[0042]
[0043] in, It is a learnable weight matrix. These are the feature dimensions of the query and the key. It is the feature dimension of the value;
[0044] Then Flatten and transpose along the spatial dimension to obtain , , Attention weight matrix Calculate the dot product of the query and the key, and divide by the scaling factor. Then, after normalization using the Softmax function, we get:
[0045]
[0046] Output of self-attention mechanism From attention weight matrix AND value Weighted summation yields:
[0047]
[0048] Then, Reshaping back to the original spatial dimensions, we obtain To preserve original information and facilitate gradient flow, the output of the self-attention module is connected to the input via a residual connection. Add:
[0049]
[0050] This self-attention module can dynamically generate context-aware feature representations for each spatial location based on the content of the input feature map, effectively modeling global dependencies; attention weights It is dynamically generated for each input image, giving the model great flexibility to adapt to complex feature relationships in different image scenarios.
[0051] Furthermore, the specific method by which the spatial attention submodule identifies and emphasizes key spatial regions in the feature map and generates spatial attention weights includes:
[0052] Feature maps enhanced by self-attention The data is fed into the spatial attention module to further identify and highlight key spatial regions in the feature map.
[0053] First, global average pooling (avg) and global max pooling (max) are performed along the channel dimension to generate two two-dimensional spatial descriptors:
[0054]
[0055]
[0056] These two descriptors are concatenated along the channel dimension to form a Feature map;
[0057] Subsequently, the stitched feature map is passed through a single The convolutional layers with convolutional kernels are processed to reduce the dimensionality to a single-channel spatial attention map;
[0058] Finally, spatial attention weights are generated using the Sigmoid function. :
[0059]
[0060] in, Represents a single Convolutional layers with convolutional kernels;
[0061] Learned spatial attention weights Its input feature map Element-wise multiplication yields the final output feature map of the channel-space self-attention module. :
[0062] .
[0063] Furthermore, the specific method for fusing the global features with the temporal embedding features generated based on the time step information in step S3 to obtain the conditional guidance features includes:
[0064] S3-1: The time step information is encoded into a high-dimensional time vector through an independent time embedding network. ;
[0065] S3-2: Represent the global features With the high-dimensional time vector Concatenate along the feature dimension;
[0066] S3-3: The concatenated fused features are processed through a linear transformation layer to generate a unified conditional feature representation. The specific formula is shown below:
[0067]
[0068] Here, Contact represents feature fusion, and Linear represents linear transformation.
[0069] Furthermore, the specific method for inputting the Gaussian noise image and conditional guided features into the denoising diffusion probability model in step S4, and performing an iterative reverse denoising process conditioned on the conditional guided features through the denoising diffusion probability model to gradually remove noise and generate the target defect sample image includes:
[0070] S4-1: Using a projection module, the dimension of the conditional features is adjusted to match the feature map of the intermediate layer of the U-Net backbone network of the denoising diffusion probability model;
[0071] S4-2: The adjusted conditional features are embedded into each residual block of the backbone network U-Net using residual injection.
[0072] S4-3: The backbone network uses noisy images The input was used, and the fusion conditions were explicitly received. The noise prediction function dynamically adjusts its internal feature representation to predict the noise to be removed in the current step, and becomes... ;in For single-step retention coefficients, The cumulative coefficient, used to calculate the mean of the reverse process, can be expressed as:
[0073]
[0074] S4-4: Based on the predicted noise, the denoised image is calculated according to the backsampling formula of the denoising diffusion probability model. The calculation formula for the backsampling process is updated as follows:
[0075]
[0076] Among them, the mean and variance Predicted by a neural network. The variance can be fixed at... Related non-learning items, thus focusing the learning emphasis on the mean. superior.
[0077] S4-5: Repeat steps S4-3 to S4-4, perform multiple iterations, gradually remove noise, and finally generate the target defect sample image.
[0078] Furthermore, the visual Transformer and the denoising diffusion probability model are jointly trained by minimizing the following loss function:
[0079]
[0080] in, Represents the original, sharp image sampled from the real data distribution; This represents a noise vector sampled from a standard normal distribution, with dimensions consistent with the image. Represents the time step for sampling from a discrete uniform distribution, where The preset total number of diffusion steps; In time step The generated noisy image depends on predefined noise scheduling parameters. For a denoising network to perform denoising on a given noisy image and time step Condition information The predicted noise component.
[0081] Furthermore, the joint training process employs the AdamW optimizer, with its exponential moving average decay rate parameters set to 0.1 and 0.99, a batch size of 8, and an initial learning rate of 0.0001.
[0082] Compared with the prior art, the present invention has the following advantages:
[0083] 1. Compared with traditional generative models, this invention can generate more realistic and detailed steel plate defect samples, especially in capturing long-distance dependencies and global context, which has significant advantages and effectively enhances the generalization ability of the defect detection model.
[0084] 2. By adding the CSSAM module during the generation process, this invention solves the shortcomings of the traditional CBAM module in spatial modeling capabilities, enhances the focus on key areas, and is able to more accurately capture details and textures in the image when dealing with complex defect structures, thereby improving the robustness and accuracy of the generated image.
[0085] 3. This invention generates defect samples by employing a diffusion process and combines the advantages of ViT and CSSAM modules, demonstrating its strong robustness in complex noise environments. Compared with existing methods, even under conditions of significant noise interference, the sample quality generated by this invention remains relatively high, further proving its application potential in practical industrial environments.
[0086] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description
[0087] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, with the same or corresponding reference numerals denoteing the same or corresponding parts, wherein:
[0088] Figure 1 A schematic diagram of the structure of the steel plate surface defect image generation method based on the denoising diffusion probability model and visual Transformer provided by the present invention;
[0089] Figure 2 The CSSAM model structure diagram provided by this invention;
[0090] Figure 3 The training defect sample image provided by this invention;
[0091] Figure 4 Generated sample images of different generation models provided by this invention;
[0092] Figure 5 This is a sample image of injected noise provided by the present invention. Detailed Implementation
[0093] The exemplary embodiments disclosed in this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art. Unless otherwise specified, the technical means used in the embodiments are conventional means well known to those skilled in the art.
[0094] Figure 1 This is a diagram of the overall model structure of the present invention, with reference to the forward diffusion process and the noise addition process. Figure 1 The ForwordDiffusion Process in the image converts the original clear image data into a single, clear image. Through a series of predefined methods, at discrete time steps The step is transformed into a latent representation of approximately isotropic Gaussian noise. Specifically, at each time step ,Towards A small amount of Gaussian noise is injected into it, and its conditional probability distribution is defined as:
[0095]
[0096] Where N is a normal distribution, It is the identity matrix. For a pre-defined variance scheduling table,
[0097] This means that the mean at the current time is the state at the previous time. Scaling by a factor Here It is a hyperparameter that determines the noise intensity at each time step; the process involves a total of... Each of the diffusion steps introduces independent and identically distributed Gaussian noise into the image, gradually destroying the image structure and generating a series of images with varying degrees of degradation; specifically, the first... The variance of the added noise is denoted as ,in And with the diffusion step The forward diffusion process gradually increases with the increase of [something]; the entire forward diffusion process can be formally defined as:
[0098]
[0099] in, For the original image, Indicates the first The noisy image generated in step 1 represents the first step. Noisy image generated step by step Noise is a standard normally distributed form; with time step... From 0 to As the amount of original information contained in an image gradually decreases and approaches zero, the amount of information in the image gradually decreases and approaches zero. hour, The distribution will be very close to a standard isotropic Gaussian distribution. At this time, the original image The structural information is almost completely masked by noise; the forward diffusion process is fixed and does not involve any learnable parameters. Its main function is to generate a series of image samples (x) at different noise levels. t These samples (x, t) will be used as training data to train the inverse denoising network, which will be discussed in detail in later chapters. The goal of this inverse network is to learn how to denoise from the noisy image x. t Predicting and removing noise from the image, or directly predicting the original image x0, thus achieving the transformation from pure noise. Generate clear images.
[0100] The reverse denoising process is as follows Figure 1The Reverse Densioning Process shown aims to learn the inverse process of forward diffusion, i.e., from a noisy image x t Gradually, a clearer image is recovered, eventually from pure Gaussian noise x T Generate the target image x0; the transformation at each step of this process. It is learned by a parameterized neural network (i.e., a denoising network); in the standard Denoising Diffusion Probabilistic Models (DDPM) framework, given x0, the posterior probability... It is solvable; however, x0 is unknown during the generation process, therefore this invention learns a neural network. To approximate the true posterior distribution; this approximate distribution is usually also chosen as a Gaussian distribution:
[0101]
[0102] Among them, the mean and variance Predicted by a neural network; and the variance can be fixed at 1. Related non-learning items, thus focusing the learning emphasis on the mean. The mean is calculated as follows:
[0103]
[0104] Here, It is the core denoising network, whose task is to predict the noise level at x given the current noisy image and time step. t Noise added to improve the denoising network; To improve the representation capabilities and quality of generated images, this paper proposes an enhanced network based on the U-Net architecture, which innovatively integrates ViT and the Channel and Spatial Self-Attention Module (CSSAM) proposed in this invention. This network aims to effectively capture global contextual information and local details of images and perform adaptive denoising based on time steps.
[0105] To capture the global dependencies of the image and enhance the model's denoising performance, we first utilize a pre-trained ViT algorithm on the current noisy image x. t Global feature extraction and modeling are performed; specifically, this process begins with input image x tThe image is adjusted to a fixed resolution preset by the ViT model using bilinear interpolation. The adjusted image is then divided into a series of fixed-size, non-overlapping image patches. Each patch is flattened and mapped to a unified embedding space using a linear transformation, forming an image patch embedding sequence. To further enhance feature representation and focus on key information regions, this image patch embedding sequence is first reshaped into a two-dimensional feature map, and then enhanced using the CSSAM module proposed in this invention. CSSAM uses a self-attention mechanism to model long-distance dependencies between channels and spatial dimensions, and generates attention weights to highlight feature regions important for denoising and suppress noise interference. The CSSAM-enhanced feature map is then flattened back into an image patch sequence, and learnable positional encodings are added to preserve spatial information. A learnable CLS token is also added to the front of the sequence. This complete sequence is then fed into a module consisting of a multi-layer Transformer encoder. Each encoder layer includes a multi-head self-attention mechanism to capture long-distance dependencies between image patches, a feedforward neural network to perform nonlinear transformations on each image patch representation, and extensive use of layer normalization and residual connections to stabilize the training process and promote efficient information flow. Finally, after processing by multiple Transformer encoders, the output of the CLS token is treated as an image. The global feature vector, denoted as Meanwhile, time step Encoded into a high-dimensional time vector via an independent temporal embedding network Finally, the extracted global image features With time embedding The features are concatenated along the feature dimension and then fused through a linear transformation layer to obtain a unified conditional feature representation. The specific formula is shown below:
[0106]
[0107] Conditions after fusion It was used to guide the denoising process of the U-Net backbone network. In order to... This fused feature effectively interacts with convolutional feature maps of different scales in U-Net. First, it is adjusted through a projection module to match the spatial dimensions and number of channels of the intermediate feature maps in U-Net. Then, the adjusted conditional feature is embedded into the individual residual blocks of U-Net via residual injection. This design allows the network to adapt to global contextual information. and current noise level By dynamically adjusting its internal feature representation, the denoising network achieves more accurate noise prediction. The input includes not only noisy images It also explicitly accepted the fusion conditions. Therefore, the noise prediction function becomes Accordingly, the formula for calculating the mean of the reverse process can be expressed as:
[0108]
[0109] Finally, the calculation formula for the backsampling process is updated as follows:
[0110]
[0111] Through the aforementioned network structure and conditional injection mechanism, this invention can more effectively utilize global information and temporal context to guide the denoising process, thereby generating higher quality and more detailed images.
[0112] Channel and Spatial Self-Attention Module (CSSAM), such as Figure 2 As shown, in order to effectively capture complex dependencies in feature maps and improve the model's ability to perceive key information, this invention proposes a Channel and Spatial Self-Attention Module (CSSAM). The core idea of CSSAM is to introduce a self-attention mechanism (SA) layer in series between the original channel attention module and spatial attention module of CBAM. This design aims to further enhance the expressive power and context awareness of features by explicitly modeling the global long-distance dependencies within feature maps, thereby enabling the network to more accurately focus on the feature information most important to the current task. Specifically, the overall structure of CSSAM includes three core components in sequence: channel attention, self-attention enhancement, and spatial attention.
[0113] The channel attention module aims to explicitly model the importance between features from different channels; given an input feature map... This module first uses global average pooling and global max pooling operations in parallel, along the spatial dimension. Aggregate information and generate two separate results. Channel computed descriptor:
[0114]
[0115]
[0116] The two descriptors are then fed into a multilayer perceptron with shared parameters; this MLP contains two fully connected layers, the first for dimensionality reduction and the second for channel recovery; the MLP output features from the two pooling paths are element-wise summed, and then the final channel attention weights are generated through a sigmoid activation function. :
[0117]
[0118] Finally, the learned channel attention weights Compared with the original input feature map Element-wise multiplication is performed to obtain the feature map after channel attention weighting. :
[0119]
[0120] Following the channel attention module, this invention introduces a self-attention mechanism to further refine the feature map. This invention aims to explicitly capture long-range dependencies between any two spatial locations within a feature map. It employs standard scaled dot product self-attention. First, the feature map after channel attention enhancement... The query, key, and value are generated through three independent linear transformations. To facilitate the computation of the dot product attention, these tensors are reshaped before attention calculation, as follows:
[0121]
[0122] in It is a learnable weight matrix. These are the feature dimensions of the query and the key. It is the feature dimension of the value; subsequently, Flatten and transpose along the spatial dimension to obtain , , Attention weight matrix Calculate the dot product of the query and the key, and divide by the scaling factor. Then, after normalization using the Softmax function, we get:
[0123]
[0124] Output of self-attention mechanism From attention weight matrix AND value Weighted summation yields:
[0125]
[0126] Then, Reshaping back to the original spatial dimensions, we obtain To preserve original information and facilitate gradient flow, this invention connects the output of the self-attention module to the input via a residual connection. Add:
[0127]
[0128] This self-attention module can dynamically generate context-aware feature representations for each spatial location based on the content of the input feature map, effectively modeling global dependencies; attention weights It is dynamically generated for each input image, giving the model great flexibility to adapt to complex feature relationships in different image scenarios.
[0129] Finally, the feature map is enhanced with self-attention. The data is fed into the spatial attention module to further identify and emphasize key spatial regions in the feature map. Similar to standard CBAM, this module first performs global average pooling and global max pooling along the channel dimension to generate two two-dimensional spatial descriptors:
[0130]
[0131]
[0132] These two descriptors are concatenated along the channel dimension to form a The feature map. Then, this stitched feature map is passed through a single... The convolutional layers with convolutional kernels are processed to reduce the dimensionality to a single-channel spatial attention map. Finally, spatial attention weights are generated using the sigmoid function. :
[0133]
[0134] Ultimately, the learned spatial attention weights will be... Its input feature map Element-wise multiplication yields the final output feature map of the CSSAM module. :
[0135]
[0136] This process enables the model to focus on the most information-rich regions in space. In summary, CSSAM achieves fine-tuning of feature maps in both channel and spatial dimensions through the cascading effects of channel attention, self-attention enhancement, and spatial attention, and incorporates global contextual information, thereby providing more discriminative and expressive feature representations for subsequent network layers.
[0137] The denoising process is formalized as a noise prediction problem, and training is performed by minimizing the difference between the model's predicted noise and the actual noise. This invention introduces ViT and CSSAM to improve the model's ability to model Gaussian degraded images, while maintaining consistency in the core loss objective. The final model's loss function can be expressed as follows:
[0138]
[0139] in, This represents the original, sharp image sampled from the real data distribution. This represents a noise vector sampled from a standard normal distribution, with dimensions consistent with the image. This represents the time step from which samples are taken from a discrete uniform distribution, where T is the preset total number of diffusion steps. It is a noisy image generated at time step t, and its generation process depends on predefined noise scheduling parameters. For a denoising network to perform denoising on a given noisy image and time step Under the given conditions, the predicted noise components.
[0140] During the training phase, this invention employs the AdamW optimization algorithm to optimize model parameters. Optimization is performed. The optimizer's exponential moving average decay rate parameter is used. and The values were set to 0.1 and 0.99 respectively. During training, the size of each mini-batch was set to 8. The initial learning rate was set to 0.0001.
[0141] Example
[0142] The NEU-CLS dataset contains six steel plate surface categories: Cracking (Cr), Inclusion (In), Patches (Pa), Pitted Surface (PS), Rolled-in Scale (RS), and Scratches (Sc). Each defect class has 300 samples. Example dataset samples are shown below. Figure 4 As shown.
[0143] Example of generating samples Figure 5 As shown in Table 1, to evaluate the quality of the generated samples, the FID (Frechet Inception Distance) score was used to compare the samples with various generative models. The comparison methods included the classic generative adversarial model structures DCGAN and LSGAN, and DDPM and DDIM using a diffusion process, totaling four methods, as well as the method of this invention. For simplicity, the method of this invention will be referred to as the DiffT method below. The experimental results are shown in Table 1:
[0144] Table 1. Results of the Experiment Comparison of Sample Quality
[0145] method Cr In Pa PS RS Sc average DCGAN 222.86 203.31 257.42 188.75 280.39 226.48 229.87 cGAN 128.48 219.54 261.99 261.08 203.51 324.08 233.11 WGAN 448.51 342.26 252.07 395.94 313.57 337.31 348.28 DDPM 176.12 155.77 161.19 157.74 145.68 197.87 165.73 DDIM 106.19 117.66 189.88 120.03 104.46 162.23 133.41 DiffT 70.44 127.68 103.21 80.49 51.45 169.36 100.44
[0146] The lower the FID score, the better the quality of the generated sample. As can be seen from Table 1, the method proposed in this invention achieved the best average FID score. Although it did not achieve the best FID results in all defect categories, the method proposed in this invention obtained the lowest FID score in more than half of the categories.
[0147] To further verify the effectiveness of the module proposed in this invention and demonstrate its ability to improve the generation quality of the generative model, an ablation experiment was conducted. The aim was to observe the changes in network performance by removing the module proposed in this invention, thereby verifying the impact of this invention on the results. The experimental results are shown in Table 2. The experimental setup included removing the CSSAM module and introducing or combining seven configurations, such as spatial attention (SAM), channel attention (CAM), and self-attention mechanism (S-AM), and comparing them with this invention.
[0148] Table 2 Ablation Experiment Results
[0149] SAM CAM S-AM Cr In Pa Ps Rs Sc AVG 142.09 146.85 128.52 110.97 105.53 195.67 138.27 √ 127.71 170.26 117.46 115.71 84.48 216.58 138.70 √ 116.37 192.16 169.26 109.98 96.78 187.56 145.35 √ 144.91 200.44 172.23 125.77 102.06 170.67 152.68 √ √ 104.67 184.75 161.87 123.64 95.68 174.04 140.78 √ √ 120.67 139.16 116.33 110.98 93.26 170.58 125.16 √ √ 90.64 138.26 110.36 93.54 84.41 176.86 115.68 √ √ √ 70.44 127.68 103.21 80.49 51.45 169.36 100.44
[0150] Experimental results show that removing the CSSAM module leads to a significant decrease in the quality of the generated images. Although the introduction of spatial attention, channel attention, or self-attention mechanisms alone shows certain advantages in some categories, the overall effect is still not ideal, and the performance fluctuates greatly across different categories. In contrast, the combined use of different attention mechanisms further improves the performance compared to a single mechanism, demonstrating good complementarity. Ultimately, this invention achieves the best average FID result in all categories, significantly outperforming other comparative methods, fully verifying the performance of the CSSAM module in improving the quality of generated images.
[0151] To verify the robustness of this invention in complex environments, noise perturbation was introduced into the original dataset to simulate data interference in real-world scenarios. Subsequently, various image generation methods were tested using the noisy dataset, and the performance changes of each method before and after the perturbation were compared. Sample images of the noisy dataset are shown below. Figure 5 As shown in Table 3, although the quality of the samples generated by this invention is somewhat reduced, the decrease in quality for most categories is still the smallest compared to other methods. This demonstrates that the robustness of this invention is superior to other methods; the experimental results are shown in Table 3.
[0152] Table 3. Results of robustness comparison experiments
[0153] method Cr In Pa PS RS Sc Avg DCGAN 364.35 423.34 363.04 350.79 438.89 339.13 379.92 Diff 141.49 220.03 105.62 162.04 158.50 112.65 150.06 cGAN 294.79 385.62 351.78 391.62 387.70 450.85 377.06 Diff 166.31 166.08 89.79 130.54 184.19 126.77 143.95 WGAN 576.29 570.15 508.77 558.02 607.28 565.54 564.18 Diff 127.78 227.89 256.70 162.08 293.71 228.23 216.07 DDPM 261.39 276.89 287.34 254.35 266.08 322.37 278.07 Diff 85.27 121.12 126.15 96.60 120.40 124.50 112.17 DDIM 200.76 230.69 256.76 264.31 269.14 310.69 255.39 Diff 94.57 113.03 66.88 144.28 164.68 148.26 121.95 DiffT 167.17 208.91 178.83 174.98 192.09 241.99 193.99 Diff 96.73 81.23 75.62 94.49 140.64 72.63 93.56
[0154] To further verify the effectiveness of the defect generation samples and demonstrate their ability to improve the accuracy of the detection model, two lightweight detection models, EfficientNet and MobileNet, were selected, and comparative experiments were conducted using generated samples from different sources. The defect generation samples came from sources including no generated samples, DCGAN, cGAN, WGAN, DDPM, DDIM, and the method proposed in this invention. Accuracy was chosen as the most commonly used evaluation metric. The experimental results are shown in Table 2.
[0155] Table 2 Comparison of Defect Detection Experiment Results
[0156] Detection model Source of generated samples Accuracy (%) No addition 91.30 DCGAN 92.61 cGAN 93.33 EfficientNet WGAN 87.26 DDPM 96.63 DDIM 97.86 DiffT 99.16 No addition 92.04 DCGAN 92.97 cGAN 94.05 MobileNet WGAN 88.12 DDPM 95.76 DDIM 97.25 DiffT 99.53
[0157] The results show that introducing generated samples significantly improves the model's classification accuracy. In particular, samples generated based on the diffusion process show a more significant improvement in accuracy compared to samples generated by the GAN method. Combined with the sample quality assessment in Table 1, it can be further found that low-quality generated samples often lead to a decline in detection performance, highlighting the crucial role of sample quality in defect detection tasks.
[0158] This invention achieved the best detection performance among all comparison methods, further demonstrating that its generated samples have high quality and can effectively enhance the model's discriminative ability, thereby significantly improving the overall performance of defect detection.
[0159] The advantages and beneficial effects of the embodiments of the present invention are as follows:
[0160] 1. Compared with traditional generative models, this invention can generate more realistic and detailed steel plate defect samples, especially in capturing long-distance dependencies and global context, which has significant advantages and effectively enhances the generalization ability of the defect detection model.
[0161] 2. By adding the CSSAM module during the generation process, this invention solves the shortcomings of the traditional CBAM module in spatial modeling capabilities, enhances the focus on key areas, and is able to more accurately capture details and textures in the image when dealing with complex defect structures, thereby improving the robustness and accuracy of the generated image.
[0162] 3. This invention generates defect samples using a diffusion process and combines the advantages of ViT and CSSAM modules, demonstrating its strong robustness in complex noisy environments. Compared to existing methods, even under conditions of significant noise interference, the sample quality generated by this invention remains relatively high, further proving its application potential in real-world industrial environments.
[0163] In summary, this invention provides a novel, simple, and rapid method for generating surface defect samples of steel plates. This method can acquire a large number of high-quality defect samples, which is helpful for training large-scale detection models and achieving high detection accuracy, thus possessing significant application value and promising development prospects.
[0164] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A method for generating images of surface defects in steel plates based on a denoising diffusion probability model and a visual Transformer, characterized in that, The method includes the following steps: Step S1: Convert the original defect image into a Gaussian noise image through multi-step noise addition, and determine its corresponding time step information; Step S2: Extract global features of the Gaussian noise image using a visual Transformer; wherein the visual Transformer integrates a channel-self-space attention module to enhance attention on features in both channel and spatial dimensions; Step S3: Fuse the global features with the temporal embedding features generated based on the time step information to obtain conditional guidance features; Step S4: Input the Gaussian noise image and the conditional guided features into the denoising diffusion probability model, and perform an iterative reverse denoising process based on the conditional guided features through the denoising diffusion probability model to gradually remove noise and generate the target defect sample image.
2. The method for generating steel plate surface defect images based on a denoising diffusion probability model and a visual Transformer according to claim 1, characterized in that, The specific method for converting the original defect image into a Gaussian noise image through multi-step noise addition and determining its corresponding time step information in step S1 includes: Original defect image In discrete time steps Transformed into a latent representation of isotropic Gaussian noise Specifically: At each time step t, towards A small amount of Gaussian noise is injected into it, and its conditional probability distribution is defined as: Where N represents a normal distribution, It is the identity matrix. A pre-defined variance scheduling table; This means that the mean at the current time is the state at the previous time. Scaling by a factor , here It is a hyperparameter that determines the noise intensity at each time step; The process involves a total of [number] steps. Each of the diffusion steps introduces independent and identically distributed Gaussian noise into the image, gradually destroying the image structure and generating a series of images with varying degrees of degradation; specifically, the first... The variance of the added noise is denoted as , in With diffusion step It gradually increases with the increase of ; the entire forward diffusion process can be formally defined as: in, For the original image, Indicates the first The noisy image generated in step 1 represents the first step. Noisy image generated step by step Noise that follows a standard normal distribution; As time goes by From 0 to As the amount of original information contained in an image gradually decreases and approaches zero, the amount of information in the image gradually decreases and approaches zero. hour, The distribution will be very close to a standard isotropic Gaussian distribution. At this time, the original image The structural information was almost completely obscured by noise.
3. The method for generating steel plate surface defect images based on a denoising diffusion probability model and a visual Transformer according to claim 1, characterized in that, Step S2 involves using a visual Transformer to extract global features from the Gaussian noise image. The visual Transformer integrates a channel-self-spatial attention module to enhance feature attention in both channel and spatial dimensions. Specific methods for this enhancement include: S2-1: Input image x t Adjust to a preset size and divide into a sequence of image blocks, then obtain an image block embedding sequence through linear mapping; S2-2: The image block embedding sequence is reshaped into a two-dimensional feature map, and then processed in series through the channel attention submodule, self attention submodule and spatial attention submodule in the channel-space self attention module; The channel attention submodule is used to model the importance of each channel feature and generate channel attention weights; The self-attention submodule is used to explicitly capture the long-distance dependency between any two spatial locations within the feature map, generating a context-aware feature representation; The spatial attention submodule is used to identify and emphasize key spatial regions in the feature map and generate spatial attention weights; S2-3: The feature map enhanced by the channel-space self-attention module is flattened back into a sequence, and positional encoding and learnable classification labels are added to form a complete input sequence; S2-4: The input sequence is fed into a module consisting of a multi-layer Transformer encoder for processing, and the output of the classification label is used as the global feature representation of the noisy image.
4. The method for generating steel plate surface defect images based on a denoising diffusion probability model and a visual Transformer according to claim 3, characterized in that, The specific methods used by the channel attention submodule to model the importance of features among channels and generate channel attention weights include: Given an input feature map The channel attention submodule first performs global average pooling (avg) and global max pooling (max) operations in parallel along the spatial dimension. Aggregate information and generate two separate results. Channel computed descriptor: Where F represents the input feature map, c represents the channel index, H and W represent the height and width of the feature map, and F(:,i,j) represents the feature value of the c-th channel at position (i,j); The two descriptors are then fed into a multilayer perceptron (MLP) with shared parameters; this MLP contains two fully connected layers, the first for dimensionality reduction and the second for channel recovery; the MLP output features from the two pooling paths are element-wise summed, and then the final channel attention weights are generated by passing them through a sigmoid activation function. : in, This represents the Sigmoid activation function; Learned channel attention weights Compared with the original input feature map Element-wise multiplication (broadcasting is used for spatial dimensions) is performed to obtain a feature map that has been weighted by channel attention. : 。 5. The method for generating steel plate surface defect images based on a denoising diffusion probability model and a visual Transformer according to claim 3, characterized in that, The self-attention submodule is used to explicitly capture long-distance dependencies between any two spatial locations within the feature map, and the specific methods for generating context-aware feature representations include: Following the channel attention module, a self-attention mechanism is introduced to further refine the feature maps. The aim is to explicitly capture long-distance dependencies between any two spatial locations within a feature map; Use standard scaled dot product self-attention; First, the feature map after channel attention enhancement. The query, key, and value are generated through three independent linear transformations; these tensors are reshaped before attention computation to facilitate the calculation of the dot product attention; specifically as follows: in, It is a learnable weight matrix. These are the feature dimensions of the query and the key. It is the feature dimension of the value; Then, Flatten and transpose along the spatial dimension to obtain , , Attention weight matrix Calculate the dot product of the query and the key, and divide by the scaling factor. Then, after normalization using the Softmax function, we get: Output of self-attention mechanism From attention weight matrix AND value Weighted summation yields: Then, Reshaping back to the original spatial dimensions, we obtain To preserve original information and facilitate gradient flow, the output of the self-attention module is connected to the input via a residual connection. Add: This self-attention module can dynamically generate context-aware feature representations for each spatial location based on the content of the input feature map, effectively modeling global dependencies; the attention weight A is dynamically generated for each input image, giving the model great flexibility to adapt to complex feature relationships in different image scenes.
6. The method for generating steel plate surface defect images based on a denoising diffusion probability model and a visual Transformer according to claim 3, characterized in that, The specific methods used by the spatial attention submodule to identify and emphasize key spatial regions in the feature map and generate spatial attention weights include: Feature maps enhanced by self-attention The data is fed into the spatial attention module to further identify and highlight key spatial regions in the feature map. First, global average pooling (avg) and global max pooling (max) are performed along the channel dimension to generate two two-dimensional spatial descriptors: These two descriptors are concatenated along the channel dimension to form a Feature map; Subsequently, the stitched feature map is passed through a single The convolutional layers with convolutional kernels are processed to reduce the dimensionality to a single-channel spatial attention map; Finally, spatial attention weights are generated using the Sigmoid function. : in, Represents a single Convolutional layers with convolutional kernels; Learned spatial attention weights Its input feature map Element-wise multiplication yields the final output feature map of the channel-space self-attention module. : 。 7. The method for generating steel plate surface defect images based on a denoising diffusion probability model and a visual Transformer according to claim 1, characterized in that, The specific method for fusing the global features with the temporal embedding features generated based on the time step information in step S3 to obtain the conditional guidance features includes: S3-1: The time step information is encoded into a high-dimensional time vector through an independent time embedding network. ; S3-2: Represent the global features With the high-dimensional time vector Concatenate along the feature dimension; S3-3: The concatenated fused features are processed through a linear transformation layer to generate a unified conditional feature representation. The specific formula is shown below: Here, Contact represents feature fusion, and Linear represents linear transformation.
8. The method for generating steel plate surface defect images based on a denoising diffusion probability model and a visual Transformer according to claim 1, characterized in that, The specific method described in step S4, which involves inputting the Gaussian noise image and conditional guided features into a denoising diffusion probability model, and performing an iterative reverse denoising process conditioned on the conditional guided features through the denoising diffusion probability model to gradually remove noise and generate a target defect sample image, includes: S4-1: Using a projection module, the dimension of the conditional features is adjusted to match the feature map of the intermediate layer of the U-Net backbone network of the denoising diffusion probability model; S4-2: The adjusted conditional features are embedded into each residual block of the backbone network U-Net using residual injection. S4-3: The backbone network uses noisy images The input was used, and the fusion conditions were explicitly received. The noise prediction function dynamically adjusts its internal feature representation to predict the noise to be removed in the current step, and becomes... ;in For single-step retention coefficients, The cumulative coefficient, used to calculate the mean of the reverse process, can be expressed as: S4-4: Based on the predicted noise, the denoised image is calculated according to the backsampling formula of the denoising diffusion probability model. The calculation formula for the backsampling process is updated as follows: Among them, the mean and variance Predicted by a neural network, where the variance can be fixed as... Related non-learning items, thus focusing the learning emphasis on the mean. superior; S4-5: Repeat steps S4-3 to S4-4, perform multiple iterations, gradually remove noise, and finally generate the target defect sample image.
9. The method for generating steel plate surface defect images based on a denoising diffusion probability model and a visual Transformer according to claim 1, characterized in that, The visual Transformer and the denoised diffusion probability model are jointly trained by minimizing the following loss function: in, Represents the original, sharp image sampled from the real data distribution; This represents a noise vector sampled from a standard normal distribution, with dimensions consistent with the image. Represents the time step for sampling from a discrete uniform distribution, where The preset total number of diffusion steps; In time step The generated noisy image depends on predefined noise scheduling parameters. For a denoising network to perform denoising on a given noisy image and time step Condition information The predicted noise component.
10. The method for generating steel plate surface defect images based on a denoising diffusion probability model and a visual Transformer according to claim 1, characterized in that, The joint training process uses the AdamW optimizer with exponential moving average decay rate parameters set to 0.1 and 0.99, a batch size of 8, and an initial learning rate of 0.0001.
Citation Information
Cited By
Runoff prediction method and device, electronic equipment and computer readable storage medium
CN122132784A