Defect Detection Method Based on Image Inpainting

Abstract

The method discloses a defect detection method based on image restoration, which only needs to train positive samples, solves the problem of imbalance between positive and negative samples in industrial production, masks the picture in blocks and uses an image restoration network to restore and reconstruct it, and then combines the reconstructed images of each masked block to obtain the final defect-free reconstructed image, thereby ensuring that the reconstruction of the abnormal part is not affected by its original value and limiting the model's ability to reconstruct abnormalities, thereby obtaining better reconstruction effects and higher defect detection accuracy.

Description

Technical Field

[0001] The invention relates to a defect detection method, in particular to a defect detection method based on image restoration. Background Art

[0002] Product surface defect detection is a very important part of industrial manufacturing and production. Its detection accuracy will directly affect the quality of the final output product.

[0003] With the development of computer vision and artificial intelligence, more and more product defect detection uses machine vision methods to replace traditional manual inspection. Many defect detection methods based on deep learning have excellent performance. However, in actual industrial manufacturing environments, the number of defect-free samples is often much larger than the number of defective samples. The imbalance of positive and negative samples will make the model training that requires a large amount of labeled data insufficient, thus affecting the detection effect.

[0004] The success of image reconstruction in recent years has influenced unsupervised anomaly detection and segmentation methods. Many reconstruction-based anomaly detection methods have been proposed, such as training autoencoders and generative adversarial networks (GANs) using only defect-free positive samples to enable them to learn the distribution of normal defect-free samples, and then identifying and segmenting defects based on the difference between the input image and its reconstructed image. However, although they are only trained based on normal samples, the reconstructed images cannot completely eliminate the defect information of abnormal images during the test process. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a defect detection method based on image restoration, and the specific technical solution is as follows:

[0006] 1. A defect detection method based on image restoration, characterized in that it comprises the following steps:

[0007] S1: Construct a training dataset based on defect-free positive samples and perform masking on the images in the dataset.

[0008] S2: Input the masked image processed by S1 into the image restoration network model for training to obtain the reconstructed image, and optimize the model by minimizing the reconstruction loss.

[0009] S3: After the defect image to be tested is masked by the masking process described in S1, it is input into the trained image restoration network to obtain a reconstructed image of the masked part.

[0010] S4: Combining the reconstructed images of all the masked parts of the defect image to be tested in sequence to obtain a defect-free reconstructed image of the entire defect image.

[0011] S5: Calculate the residual image and SSIM evaluation image between the defect image to be tested and its reconstructed image, perform threshold segmentation on the two respectively, and then combine them to obtain the final defect recognition and positioning results.

[0012] Furthermore, the steps of performing masking processing on the image in step S1 are as follows:

[0013] S1.1: Divide the input image into n×n grid images with image blocks of the same size, where n can be set according to actual needs.

[0014] S1.2: Generate a mask with the same size as the input image, wherein the mask is a binary mask whose value is 1 in the masked area and 0 in other areas.

[0015] S1.3: Obtain n×n masked images by masking a grid with the input image and mask in sequence. The specific steps are as follows:

[0016] The input image x is divided into n×n grids of equal size after step S1.1. The grid x at (i, j) (i,j) It can be expressed as x·m (i,j) , where i,j∈{1,2.....n}, m(i,j) represents the mask m at x (i,j) The value of the area is 1, and the other areas are 0, so the mask image can be obtained by x·(1-m (i,j) )get.

[0017] Furthermore, the image restoration network in step S2 is an encoder-decoder network based on U-net with skip connections, which introduces skip connections to connect the features of each encoder layer with the features of the decoder layer of the corresponding layer, thereby ensuring fine restoration and reconstruction.

[0018] Furthermore, the optimization target in the image restoration network training process in step S2 is the reconstruction loss of the masked part and the non-masked part, which is obtained by the mean square sum loss and the structural similarity SSIM loss. The specific design details are as follows:

[0019] L re =βL unm +(1-β)L m (1)

[0020] Where β is the loss balance hyperparameter, L re The overall reconstruction loss, L unm is the reconstruction loss of the unmasked area, L m is the reconstruction loss of the masked area.

[0021]

[0022] In the formula, is the reconstructed image obtained by inputting x through the image restoration network, α is the loss balance hyperparameter, m is the mask, and & represents bitwise multiplication.

[0023] The mean square error (MSE) is defined as:

[0024]

[0025] The SSIM formula can be expressed as:

[0026]

[0027] Where η x , η y are the average grayscale values ​​of the images x and y to be compared, σ x , σ y are the grayscale standard deviations of x and y, σ xy is the covariance between x and y, c1 and c2 are constants used to maintain stability.

[0028] Furthermore, the specific steps of step S4 are as follows:

[0029] For the input image, the reconstruction of each masked part can be expressed as: G(x·(1-m (i,j) ))·m (i,j) , where G represents the image restoration network. Combining the reconstruction images of all the masked parts in turn can obtain the defect-free reconstruction image x′ of the entire image, which is expressed as:

[0030]

[0031] Furthermore, the implementation process of step S5 is as follows:

[0032] The reconstruction residual value of the defect-free input image and its reconstructed image in the training set is defined as:

[0033]

[0034] Then the residual image segmentation threshold of the input image to be tested and its reconstructed image can be determined as the maximum value of the reconstructed residual of the training sample, that is:

[0035] T1=maxv (8)

[0036] The residual binary image of the input to be tested and its reconstruction is:

[0037]

[0038] The value of 0 represents a defective pixel. The structural similarity between the defect-free input image and the reconstructed image in the training set at the window with (i, j) as the center point is calculated by formula (5) and defined as S (i,j), then the segmentation threshold of the SSIM evaluation image can be determined as:

[0039]

[0040] That is, the mean of the structural similarity of all windows. The binary image of the SSIM evaluation image is:

[0041]

[0042] Where 0 represents a defect. The final defect segmentation result is:

[0043] I result =I residual &I ssim (12)

[0044] The beneficial effects of the present invention are as follows:

[0045] This method only needs to use defect-free samples for unsupervised training. There is no need to collect a large number of defective samples, nor does it need to spend a lot of time and manpower on labeling. It is of practical significance to solve the problem that defective samples are difficult to collect and defect labeling is time-consuming and labor-intensive in actual scenarios.

[0046] The image is divided into blocks and masked and then repaired and reconstructed by the image inpainting network, which suppresses the model's ability to reconstruct abnormalities and ensures that the reconstruction of the abnormal part is not affected by its original value, thereby obtaining better reconstruction effect and higher defect detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of the defect detection method based on the image restoration network of the present invention.

[0048] Figure 2 This is a structural diagram of the image restoration network used in the present invention.

[0049] Figure 3 This is a diagram of the defect repair and reconstruction process described in the present invention. DETAILED DESCRIPTION

[0050] The present invention will be described in detail below according to the accompanying drawings and preferred embodiments, and the purpose and effect of the present invention will become more clear. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0051] like Figure 1As shown, the defect detection method based on image restoration of the present invention includes the following steps: dividing the input image to be tested into grid images with image blocks of the same size; generating a mask of the same size as the input image; obtaining masked images in turn by masking one grid each time with the input image and the mask; sending the masked image to a trained image restoration network for restoration and reconstruction to obtain a reconstructed image of each grid, and combining the reconstructed images of each grid in turn to obtain a complete defect-free reconstructed image; calculating the residual threshold segmentation image of the input and reconstructed images and the threshold image of the SSIM evaluation image, and combining the two to obtain the final defect detection result image. The present invention has a good defect-free reconstruction effect on defective samples, high defect detection accuracy and high practical application value.

[0052] As an example, the specific steps of the defect detection method based on image restoration of the present invention are as follows:

[0053] S1: Construct a training dataset based on defect-free positive samples and perform masking on the images in the dataset.

[0054] In this example, the image size of the dataset is 256×256, and the mask processing steps are as follows:

[0055] S1.1: Divide the input image into 4×4 grid images of size 64×64.

[0056] S1.2: Generate a mask with the same size as the input image, 256×256. The mask is a binary mask, whose value is 1 in the masked area and 0 in other areas.

[0057] S1.3: Obtain a 4×4 masked image by masking a grid with the input image and mask in sequence. The specific steps are as follows:

[0058] The input image x is divided into 4×4 grids of equal size after step S1.1. The grid x at (i, j) (i,j) It can be expressed as x·m (i,j) , where i,j∈{1,2.....4}, m(i,j) represents the mask m at x (i,j) The value of the area is 1, and the other areas are 0, so the mask image can be obtained by x·(1-m (i,j) )get.

[0059] S2: Input the masked image processed by S1 into the image restoration network model for training to obtain the reconstructed image, and optimize the model by minimizing the reconstruction loss.

[0060] S2.1 As Figure 2As shown in the figure, the image restoration network is an encoder-decoder network based on U-net with skip connections. During the encoding process, the image is first down-sampled four times after being input into the network. Each time the image dimension is halved and the number of channels is doubled. During the decoding process, the reconstructed image is obtained through four corresponding upsampling operations. The use of skip connections fuses the feature map of the encoding process with the feature map of the corresponding layer during decoding, so that the decoder can obtain more high-resolution information when upsampling, and then more perfectly restore the detail information in the original image, thereby ensuring more refined reconstruction.

[0061] The process of image restoration and reconstruction by the image restoration network described in S2.2 is as follows Figure 3 As shown, the optimization target in the network training process is the reconstruction loss of the masked part and the non-masked part, which is obtained by the traditional mean square error and the structural similarity SSIM error. The specific design details are as follows:

[0062] L re =βL unm +(1-β)L m (1)

[0063] Where β is the loss balance hyperparameter, L unm is the reconstruction loss of the unmasked area, L m is the reconstruction loss of the masked area.

[0064]

[0065] In the formula, is the reconstructed image obtained by inputting x through the image restoration network, α is the loss balance hyperparameter, and & represents bitwise multiplication.

[0066] The mean square error (MSE) is defined as:

[0067]

[0068] The SSIM formula can be expressed as:

[0069]

[0070] Where η x , η y are the average grayscale values ​​of the images x and y to be compared, σ x , σ y are the grayscale standard deviations of x and y, σ xy is the covariance between x and y, c1 and c2 are constants used to maintain stability.

[0071] S3: After the defect image to be tested is masked by the masking process described in S1, it is input into the trained image restoration network to obtain a reconstructed image of the masked part.

[0072] S4: Combining the reconstructed images of all the masked parts of the defect image to be tested in sequence to obtain a defect-free reconstructed image of the entire defect image.

[0073] The specific steps are as follows:

[0074] For the input image, the reconstruction of each masked part can be expressed as: G(x·(1-m (i,j) ))·m (i,j) , Combining the reconstruction images of all the masked parts in sequence can obtain the defect-free reconstruction image x′ of the entire image, which can be expressed as:

[0075]

[0076] S5: Calculate the residual image and SSIM evaluation image between the defect image to be tested and its reconstructed image, perform threshold segmentation on the two respectively, and then combine them to obtain the final defect recognition and positioning results.

[0077] The implementation process is as follows:

[0078] The reconstructed residual value of the defect-free input image and its reconstructed image is defined as:

[0079] v=(||xx′|| 2 ) 1 / 2 (7)

[0080] Then the input and reconstructed residual image segmentation threshold can be determined as the maximum value of the reconstructed residual of the defect-free image, that is:

[0081] T1=maxv (8)

[0082] The input and reconstructed residual binary images are:

[0083]

[0084] The value of 0 represents a defective pixel. The structural similarity between the defect-free input image and the reconstructed image at the window with (i, j) as the center point is calculated by formula (5) and defined as S (i,j) , then the segmentation threshold of the SSIM evaluation image can be determined as:

[0085]

[0086] That is, the mean of the structural similarity of all windows. The binary image of the SSIM evaluation image is:

[0087]

[0088] Where 0 represents a defect. The final defect segmentation result is:

[0089] I result =I residual &I ssim (12)

[0090] This method improves the problem that the traditional reconstruction-based defect detection method cannot completely avoid the reconstructed defective pixels during testing, and has better defect-free reconstruction capability and higher defect detection accuracy.

[0091] Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention shall be included in the protection scope of the invention.

Claims

1. A defect detection method based on image restoration, characterized in that: The following steps are involved: S1: Construct a training dataset based on defect-free positive samples and perform masking on the images in the dataset; S2: Input the masked image processed by S1 into the image restoration network model for training to obtain the reconstructed image, and optimize the model by minimizing the reconstruction loss; S3: After the defect image to be tested is processed by the mask described in S1, it is input into the trained image restoration network to obtain a reconstructed image of the masked part; S4: sequentially combining all the masked part reconstruction images of the defect image to be tested to obtain a defect-free reconstruction image of the entire defect image; S5: Calculate the residual image and SSIM evaluation image between the defect image to be tested and its reconstructed image, perform threshold segmentation on the two respectively, and then combine them to obtain the final defect recognition and positioning results; The specific steps of performing masking processing on the images in the data set in step S1 are as follows: S1.1: Divide the input image into n×n grid images with image blocks of the same size, where n can be set according to actual needs; S1.2: Generate a mask with the same size as the input image, wherein the mask is a binary mask with a value of 1 in the masked area and a value of 0 in other areas; S1.3: Obtain n×n masked images by masking a grid with the input image and mask in sequence. The specific steps are as follows: The input image x is divided into n×n grids of equal size after step S1.

1. The grid x at (i, j) (i,j) It can be expressed as x·m (i,j) , where i,j∈{1,2,...,n}, m(i,j) represents the mask m at x (i,j) The value of the area is 1, and the value of other areas is 0, so the mask image can be obtained by x·(1-m (i,j) )get; The optimization target in the image restoration network training process in step S2 is the reconstruction loss of the masked part and the non-masked part, which is obtained by the mean square loss and the structural similarity SSIM loss. The specific design details are as follows: L re =βL unm +(1-β)L m (1) Where β is the loss balance hyperparameter, L re is the overall reconstruction loss, L unm is the reconstruction loss of the unmasked area, L m is the reconstruction loss of the masked area; In the formula, is the reconstructed image obtained by input x through the image restoration network, α is the loss balance hyperparameter, m is the mask, and & represents bitwise multiplication; The mean square error (MSE) is defined as: The SSIM formula can be expressed as: Where η x , η y are the average grayscale values ​​of the images x and y to be compared, σ x , σ y are the grayscale standard deviations of the images x and y to be compared, σ xy is the covariance between x and y, c1 and c2 are constants used to maintain stability; The specific steps of step S4 are as follows: For the input image, the reconstruction of each masked part can be expressed as: G(x·(1-m (i,j) ))·m (i,j) , where G represents the image restoration network. Combining the reconstruction images of all the masked parts in turn can obtain the defect-free reconstruction image x′ of the entire image, which is expressed as: The implementation process of step S5 is as follows: The reconstruction residual value of the defect-free input image and its reconstructed image in the training set is defined as: Then the residual image segmentation threshold of the input image to be tested and its reconstructed image can be determined as the maximum value of the reconstructed residual of the training sample, that is: T1=max v (8) The residual binary image of the input to be tested and its reconstruction is: Among them, the value of 0 represents a defective pixel. The structural similarity between the defect-free input image and the reconstructed image in the training set at the window with (i, j) as the center point is calculated by formula (5) and is defined as S (i,j) , then the segmentation threshold of the SSIM evaluation image can be determined as: That is, the mean of the structural similarity of all windows. The binary image of the SSIM evaluation image is: Among them, 0 represents a defect; the final defect segmentation result is: I result =I residual &I ssim (12)。 2. The defect detection method based on image restoration according to claim 1, characterized in that: The image restoration network in step S2 is an encoder-decoder network based on U-net with skip connections. Skip connections are introduced to connect the features of each encoder layer with the features of the decoder layer of the corresponding layer, thereby ensuring fine restoration and reconstruction.

Images