Industrial product defect detection method based on image diffusion

Abstract

The invention discloses an industrial product defect detection method based on image diffusion, and the method comprises the steps: carrying out the image diffusion of an input picture in a training stage, inputting the diffused picture into a multi-output diffusion model, comparing the output of the multi-output diffusion model with the diffused picture, calculating a loss value, and updating the model parameters; in the testing stage, the industrial product picture is input into the diffusion model to obtain an intermediate prediction result, then the intermediate prediction result serves as the diffusion model to be input and circulated for multiple times, final output is obtained, and the number of cycles of the industrial product picture is predicted through a dynamic step number prediction module; the method can be used for carrying out defect detection on industrial product pictures and completing detection and segmentation tasks at the same time.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence image processing, and specifically relates to a method for industrial product defect detection based on industrial product image samples. Background Art

[0002] Defect detection of industrial products is an important issue in industrial production. In the past, it relied on time-consuming and labor-intensive manual inspection, but now it has gradually used automated inspection based on computer vision technology to improve efficiency. Defect detection of industrial products mainly relies on unsupervised methods, including two types of methods based on feature embedding and reconstruction. Most methods based on feature embedding use robust pre-trained models for feature extraction, which can better obtain high-level semantic features of images and help achieve higher image-level detection accuracy. Although the reconstruction-based method limits the detection capability at the image level, it can identify defects through pixel-level comparisons and perform image-level defect detection on this basis, and often has better defect location capabilities.

[0003] Reconstruction-based methods generally reconstruct images by training encoders and decoders, and assume that the generated images contain only normal features and no defect features. In this way, when the reconstructed image is significantly different from the original image, it is considered that the original image has defects, and the defects are located by comparing the differences between the two. However, it is very difficult to retain only normal features during the reconstruction process. Many algorithms will also reconstruct defect features during reconstruction, resulting in low final detection accuracy, especially when the defect area is very small.

[0004] Diffusion model generally refers to Denoising Diffusion Probabilistic Model (DDPM), which is a generative model that has emerged in the past two years. It is a parameterized Markov chain that can transform simple distributions into complex target distributions in a limited number of steps. Diffusion model generally includes two processes: forward diffusion and reverse denoising. In the forward diffusion process, Gaussian noise is gradually added to the input image. When the number of steps is large enough, the image will be completely converted into a Gaussian noise image; in the reverse denoising process, the noise is gradually removed, and the transformation from the Gaussian noise image to the target image can be achieved. The diffusion model can realize the mutual conversion of image and noise. Based on this feature, defects can be hidden in the noise and eliminated, so as to perform defect detection. Summary of the invention

[0005] The present invention comprehensively utilizes the idea of ​​diffusion model to reconstruct the industrial product image and designs an industrial product defect detection method based on image diffusion to realize defect detection.

[0006] The technical solution adopted by the present invention to solve the technical problem is: an industrial product defect detection method based on image diffusion, comprising the following steps:

[0007] S1, obtain industrial product images collected by industrial cameras to build a training data set, and select public data as the texture data set;

[0008] S2, training phase, image diffusion is performed on the input image: a normal image N of industrial products is selected from the training data set, a defect mask M is obtained according to the Perlin noise generation algorithm, and a defect image A is obtained by combining the additional texture data set image and the defect mask M. The normal image N, the defect image A and the defect mask M are used as input images, and the image diffusion is performed using the following formula to obtain the defect image. In the formula is the bitwise inversion of M, ⊙ is the dot product operation, α t Negatively correlated with the diffusion degree of defect image A, the calculation method is: β i ∈(0,1) is the diffusion degree change schedule, which is set to a linear value from 0.0001 to 0.01.

[0009] S3, input the diffused image into the multi-output diffusion model, and the multi-output diffusion model outputs the predicted normal image N_pred, the predicted defect image A_pred and the predicted defect mask M_pred respectively, wherein the output head of the predicted defect mask M_pred is composed of a group normalization layer Group Norm, a SiLU layer, a convolution layer and a Softmax layer, and the output heads of the predicted normal image N_pred and the predicted defect image A_pred are composed of a single convolution layer;

[0010] S4, compare the output of the multi-output diffusion model with the diffused image, calculate the loss value and update the model parameters;

[0011] S5, testing phase, input the industrial product image into the multi-output diffusion model to obtain the intermediate prediction result, and then use the intermediate prediction result as the input of the multi-output diffusion model for several cycles, and finally output the industrial product defect image.

[0012] Furthermore, in the step S3, a mask supervision information enhancement mask M_aug is added at the input of the multi-output diffusion model, which is obtained by changing the Perlin noise threshold of the defect mask M: the Perlin noise map m is obtained according to the Perlin noise generation algorithm, and the threshold η is set to 0.5 to obtain the defect mask M; the threshold is kept unchanged with a probability of 20%, and the threshold is increased and decreased with a probability of 40% respectively, and η=0.5±Δ is taken. Similarly, pixels with values ​​greater than η are retained, and the rest are set to 0 to obtain the enhanced mask M_aug.

[0013] Furthermore, when calculating the loss value in step S4, the loss function L is defined as follows:

[0014] L=L normal +L defect +L mask +L recon ,

[0015] The loss function L consists of four parts: normal loss L normal , defect loss L defect , mask loss L mask and the reconstruction loss L recon :

[0016] L normal =SSIM(N pred ,N)+L1(N pred ,N),

[0017] L defect =L2(A pred ,A),

[0018] L mask =γL Focal (M pred ,M)+Smooth L1 (M pred ,M),

[0019]

[0020] Normal loss L normal From the normal picture N and the predicted normal picture N pred The SSIM loss and L1 loss between them constitute the defect loss L defect Defect image A and predicted defect image A pred L2 loss between; mask loss L mask Defect mask M and predicted defect mask M pred The Focal loss and smooth L1 loss between them are composed, γ is a hyperparameter; the reconstruction loss L recon is the reconstruction sample and its truth value x i L2 loss between, reconstructed samples According to the reconstruction method, the first three loss functions are normal loss functions L normal , defect loss function L defect And the mask loss function L mask They are used to train the three output heads of the multi-output diffusion model. The fourth loss function is the reconstruction loss function L recon Used to ensure the effectiveness of reconstruction during the testing phase.

[0021] Furthermore, in step S5, the multi-output diffusion model uses hidden layer features to segment the image and outputs predicted normal images and defective images at the same time, which is used to reconstruct the image multiple times in the test phase to obtain more accurate segmentation results: Assume that the industrial product image is a defective image x t , initial mask M t+1 is a full-zero image. The two are concatenated in the channel dimension and sent to the multi-output diffusion model to obtain the predicted normal image N at time step t. t , Prediction defect image A t and the predicted defect mask M t ; A multi-step denoising process similar to DDIM is used to obtain the final defect mask M0 through multi-step reconstruction. In the training stage, the image diffusion formula is used to obtain Then subtract it from the image diffusion formula to get In the test phase, the normal image N, defect image A and defect mask M can be predicted by the multi-output diffusion model to obtain the corresponding time step prediction value N t , A t and M t , so according to x t To estimate x t-i The value of

[0022]

[0023] Where i = 1, 2, ..., t, ti is the step length, and a number that is divisible by T is selected. In the experiment, T is 1200; the predicted defective image and defect mask M t Then it is sent to the multi-output diffusion model, and the predicted mask is improved through multiple cycles of iteration to obtain the final defect mask M0.

[0024] Furthermore, in step S5, the number of cycles of the industrial product image is predicted by the dynamic step number prediction module: the KNN algorithm is used to obtain k nearest neighbor points for all images in the training data set, and the average distance between these images and the k nearest neighbor points is obtained, and the average distance is divided into n equidistant intervals S = [S1, S2, ..., S n ], each equidistant interval corresponds to a reconstruction step, and the sequence of reconstruction steps is s = [s1, s2, …, s n ]; find the average distance between the industrial product image and the k nearest neighbor points, and when the distance falls into the interval S i When , the reconstruction step number s is taken i , that is, rebuild the cycle s i times; let the reconstruction step length be l i =t / s i , then the predicted defect mask sequence is finally obtained And predict the normal picture N0, which is used to calculate the final output of the multi-output diffusion model. The final output M of the model is calculated based on the results obtained in the above steps. output , the calculation formula is in To predict the defect mask sequence M p All predicted defect masks in are bitwise averaged, ||N0-x t || is to predict the normal image N0 and the image to be detected x t The L1 distance of , λ is the weight hyperparameter used to balance the weight between the predicted defect mask and the reconstruction error, f is the mean filter of size 7*7, and * is the convolution operation.

[0025] The beneficial effects of the present invention are as follows: the present invention can be used to perform defect detection on industrial product images, and simultaneously complete detection and segmentation tasks; for detection, it can be determined whether the input image contains defects, so as to choose whether to accept the product; for segmentation, the defective area in the image can be accurately divided, so as to choose whether to repair the product on this basis, and choose to accept or discard the product. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of the detection method of the present invention;

[0027] Figure 2 is a flow chart of the training phase of the present invention;

[0028] Figure 3 Flow chart of the testing phase of the present invention. DETAILED DESCRIPTION

[0029] The specific implementation of the present invention is described below in conjunction with the accompanying drawings and examples so that those skilled in the art can better understand the present invention.

[0030] Reference Figure 1 As shown, the present invention discloses an industrial product defect detection method based on image diffusion, and the overall process of the method is divided into two parts: a training phase and a testing phase, as shown below.

[0031] S1, obtain industrial product images collected by industrial cameras to build a training data set, and select public data as the texture data set.

[0032] S2, reference Figure 2 As shown, during the training phase, image diffusion is performed on the input image.

[0033] Image diffusion is in contrast to noise diffusion in traditional diffusion models. It refers to gradually adding an image of one distribution to an image of another distribution until the image is completely transformed. This process is similar to the noise diffusion process. The input of image diffusion includes a normal image N, a defect image A, and a defect mask M. The normal image N comes from the training dataset, the defect mask M is obtained according to the Perlin noise generation algorithm, and the defect image A is obtained by combining an additional texture dataset image and the defect mask M. The image diffusion formula is defined as in is the bitwise inversion of M; ⊙ is the dot multiplication operation; α t Negatively correlated with the diffusion degree of the defect image, the calculation method is: β i ∈(0,1) is the diffusion degree change schedule, which is set to a linear value from 0.0001 to 0.01. Let α i =1-β i , x t It is a defective image obtained by image diffusion.

[0034] S3, inputs the diffused image into the modified diffusion model, i.e., the multi-output diffusion model.

[0035] The present invention transforms the output structure of the conventional diffusion model. The original output is the predicted noise, which is now changed to three outputs, namely the predicted normal image N_pred, the predicted defect image A_pred and the predicted defect mask M_pred.

[0036] The multi-output diffusion model outputs the predicted normal image N_pred, the predicted defect image A_pred and the predicted defect mask M_pred respectively, where the output head of the predicted defect mask M_pred is composed of a group normalization layer Group Norm, a SiLU layer, a convolution layer and a Softmax layer, and the output heads of the predicted normal image N_pred and the predicted defect image A_pred are composed of a single convolution layer.

[0037] In order to further improve the segmentation ability of the method, additional mask supervision information enhancement mask Maug is added at the input of the multi-output diffusion model, which is obtained by changing the Perlin noise threshold of the defect mask M.

[0038] Specifically, the Perlin noise map m is obtained according to the Perlin noise generation algorithm, and the threshold η is set to 0.5, that is, some pixels with values ​​greater than 0.5 are retained, and the rest are set to 0, to obtain the defect mask M; the threshold is kept unchanged with a probability of 20%, and the threshold is increased and decreased with a probability of 40% respectively, and η is taken as 0.5±Δ. Similarly, pixels with values ​​greater than η are retained, and the rest are set to 0, to obtain the enhanced mask Maug.

[0039] By slightly increasing or decreasing the Perlin noise threshold, an enhanced mask Maug that is slightly larger or smaller than the defect mask M can be obtained, so that the multi-output diffusion model can use it as a guide for segmentation and improve segmentation accuracy while not being overly dependent on it, and it is still useful in the testing phase.

[0040] S4, compare the output of the multi-output diffusion model with the diffused image, calculate the loss value and update the model parameters. When calculating the loss value, define the loss function L = L normal +L defect +L mask +L recon .

[0041] The loss function L consists of four parts: normal loss L normal , defect loss L defect , mask loss L mask and the reconstruction loss L recon :

[0042] L normal =SSIM(N pred ,N)+L1(N pred ,N),

[0043] L defect =L2(A pred ,A),

[0044]

[0045]

[0046] Normal loss L normal From the normal picture N and the predicted normal picture N pred The SSIM loss and L1 loss between them are composed. Since normal images pay more attention to detail features, SSIM loss is used instead of L2 loss. At the same time, L1 loss is used to reduce the sensitivity to outliers. defect Defect image A and predicted defect image A pred L2 loss between; mask loss L mask Defect mask M and predicted defect mask M pred The Focal loss and smooth L1 loss between them are used to balance the learning of difficult samples while reducing the sensitivity to outliers. It belongs to the conventional setting, and γ is a hyperparameter; the reconstruction loss L recon is the reconstruction sample and its truth value x i L2 loss between, reconstructed samples According to the reconstruction method, the first three loss functions are used to train the three output heads of the model, and the fourth loss function is used to ensure the reconstruction effect in the test phase.

[0047] S5, refer to Figure 3 As shown, in the testing phase, the industrial product image is input into the multi-output diffusion model to obtain an intermediate prediction result, and then the intermediate prediction result is used as the input of the multi-output diffusion model for several cycles, and finally the industrial product defect image is output.

[0048] The multi-output diffusion model uses the learned hidden layer features to segment the image and outputs predicted normal images and defect images at the same time, which are used to reconstruct the image multiple times during the testing phase to obtain more accurate segmentation results.

[0049] The multi-output diffusion model uses the learned hidden layer features to segment the image and outputs predicted normal images and defect images at the same time, which are used to reconstruct the image multiple times during the testing phase to obtain more accurate segmentation results.

[0050] Multi-step reconstruction: Assume that the industrial product image is a defective image x t , initial mask M t+1 is a full-zero image. The two are concatenated in the channel dimension and sent to the multi-output diffusion model to obtain the predicted normal image N at time step t. t , Prediction defect image A t and the predicted defect mask M t .

[0051] In order to enhance the segmentation effect of the model, a multi-step denoising process similar to DDIM is adopted, and the final defect mask M0 is obtained through multi-step reconstruction.

[0052] In the training stage, the image diffusion formula is used to obtain Subtract the image diffusion formula from the above formula to get In the test phase, the normal image N, defect image A and defect mask M can be predicted by the multi-output diffusion model to obtain the corresponding time step prediction value N t , A t and M t , so according to x t To estimate x t-i The value of Where i = 1, 2, ..., t, ti is the step length, and a number that is divisible by T is selected. In the experiment, T is 1200. and defect mask M t Then it is sent to the diffusion model, and the prediction mask is improved through multiple cycles and iterations to finally obtain the output M0.

[0053] In step S5, the number of cycles of the industrial product image is predicted by a dynamic step number prediction module.

[0054] Dynamic step prediction: Since images with different degrees of defects require different reconstruction steps, a dynamic step prediction module is designed to predict the reconstruction steps of different images. According to the experimental results, images with smaller defects often require fewer reconstruction steps, while images with larger defects require more reconstruction steps to make the final generated mask more accurate. Therefore, the industrial product images are compared with the normal images in the training data set, and it is believed that images with lower similarity may have larger defect areas.

[0055] Specifically, the KNN algorithm is used to obtain k nearest neighbor points for all images in the training data set, and the average distance between these images and the k nearest neighbor points is obtained. According to this average distance, the images are divided into n equidistant intervals S = [S1, S2, ..., S n ], each equidistant interval corresponds to a reconstruction step, and the sequence of reconstruction steps is s = [s1, s2, …, s n In the test phase, the average distance between the industrial product image and the k nearest neighbor points is obtained. When the distance falls into the interval S i When , the reconstruction step number s is taken i , that is, rebuild the cycle s i times. Let the reconstruction step length l i =t / s i , then the predicted defect mask sequence is finally obtained And the predicted normal image N0 is used to calculate the final output of the multi-output diffusion model.

[0056] According to the results obtained in the above steps, the final output M of the model is calculated. output , calculation formula:

[0057]

[0058] in To predict the defect mask sequence M p All predicted defect masks in are bitwise averaged, ||N0-x t || is to predict the normal image N0 and the image to be detected x t The L1 distance of , λ is the weight hyperparameter used to balance the weight between the predicted defect mask and the reconstruction error, f is the mean filter of size 7*7, and * is the convolution operation.

[0059] The above embodiments are only illustrative of the principles and effects of the present invention, as well as some embodiments of its application. For those skilled in the art, several modifications and improvements may be made without departing from the creative concept of the present invention, and all of these belong to the protection scope of the present invention.

Claims

1. An industrial product defect detection method based on image diffusion, characterized in that: The following steps are included S1, obtain industrial product images collected by industrial cameras to build a training data set, and select public data as the texture data set; S2, select a normal image N of industrial products from the training data set, obtain a defect mask M according to the Perlin noise generation algorithm, and obtain a defect image A by combining the texture data set image and the defect mask M. Take the normal image N, the defect image A and the defect mask M as input images, and use the following formula for image diffusion to obtain the defect image In the formula is the bitwise inversion of M, ⊙ is the dot product operation, α t Negatively correlated with the diffusion degree of defect image A; S3, input the diffused image into the multi-output diffusion model, and the multi-output diffusion model outputs the predicted normal image N_pred, the predicted defect image A_pred and the predicted defect mask M_pred respectively, wherein the output head of the predicted defect mask M_pred is composed of a group normalization layer Group Norm, a SiLU layer, a convolution layer and a Softmax layer, and the output heads of the predicted normal image N_pred and the predicted defect image A_pred are composed of a single convolution layer; S4, compare the output of the multi-output diffusion model with the diffused image, calculate the loss value and update the model parameters; S5, input the industrial product image into the multi-output diffusion model to obtain an intermediate prediction result, and then use the intermediate prediction result as the input of the multi-output diffusion model for several cycles, and finally output the industrial product defect image.

2. The industrial product defect detection method based on image diffusion according to claim 1 is characterized in that: The step S3 adds a mask supervision information enhancement mask M_aug at the input of the multi-output diffusion model, which is obtained by changing the Perlin noise threshold of the defect mask M: the Perlin noise map m is obtained according to the Perlin noise generation algorithm, and the threshold η is set to 0.5 to obtain the defect mask M; the threshold is kept unchanged with a probability of 20%, and the threshold is increased and decreased with a probability of 40% respectively, and η is taken as 0.5±Δ, and pixels with values ​​greater than η are retained to obtain the enhancement mask M_aug.

3. The industrial product defect detection method based on image diffusion according to claim 2 is characterized in that: When calculating the loss value in step S4, the loss function L = L normal +L defect +L mask +L recon ; Normal loss function L normal =SSIM(N pred ,N)+L1(N pred ,N), composed of normal picture N and predicted normal picture N pred The SSIM loss and L1 loss between them; the defect loss function L defect =L2(A pred ,A), is the defect image A and the predicted defect image A pred L2 loss between; mask loss function Defect mask M and predicted defect mask M pred The Focal loss and smooth L1 loss between them, γ is a hyperparameter; reconstruction loss function is the reconstruction sample and its truth value x i L2 loss between; normal loss function L normal , defect loss function L defect And the mask loss function L mask They are used to train the three output heads of the multi-output diffusion model and reconstruct the loss function L recon Used to ensure the effectiveness of reconstruction during the testing phase.

4. The industrial product defect detection method based on image diffusion according to claim 3 is characterized in that: In step S5, the multi-output diffusion model uses the hidden layer features to segment the image, and outputs the predicted normal image and defective image at the same time, and reconstructs the image multiple times to obtain a more accurate segmentation result: Let the industrial product image be the defective image x t , initial mask M t+1 is a full-zero image. The two are concatenated in the channel dimension and sent to the multi-output diffusion model to obtain the predicted normal image N at time step t. t , Prediction defect image A t and the predicted defect mask M t ; A multi-step denoising process is used to obtain the final defect mask M0 through multi-step reconstruction: obtained by the image diffusion formula Then subtract the image diffusion formula from this formula to get The normal image N, defect image A and defect mask M are predicted by the multi-output diffusion model at the corresponding time step. t , A t and M t , according to x t To estimate x t-i The value of Where i = 1, 2, ..., t, ti is the step length, and a number that is divisible by T is selected; the predicted defective image is and defect mask M t Then it is sent to the multi-output diffusion model, and the predicted mask is improved through multiple cycles of iteration to obtain the final defect mask M0.

5. The industrial product defect detection method based on image diffusion according to claim 1, 2, 3 or 4, characterized in that: In step S5, the number of cycles of the industrial product image is predicted by the dynamic step prediction module: the KNN algorithm is used to obtain k nearest neighbor points for all images in the training data set, and the average distance between the image and the k nearest neighbor points is obtained. The average distance is divided into n equidistant intervals S = [S1, S2, ..., S n ], each equidistant interval corresponds to a reconstruction step, and the sequence of reconstruction steps is s = [s1, s2, …, s n ]; find the average distance between the industrial product image and the k nearest neighbor points, when the distance falls into the interval S i When , the cycle s is rebuilt i times; let the reconstruction step length be l i =t / s i , then the predicted defect mask sequence is finally obtained And predict the normal image N0, the final output of the multi-output diffusion model in To predict the defect mask sequence M p All predicted defect masks in are bitwise averaged, ||N0-x t || is to predict the normal image N0 and the image to be detected x t The L1 distance of , λ is the weight hyperparameter, f is the mean filter of size 7*7, and * is the convolution operation.

Images