A method and device for defect detection based on a self-supervised restoration method and a storage medium
By employing a self-supervised defect detection method, and utilizing a priori masking and deformable attention mechanisms, pseudo-anomaly images are generated and preliminary localization is performed. This solves the problems of high false negative rate and low accuracy in defect detection in existing technologies, and achieves more efficient anomaly detection.
Patent Information
- Application Number
- CN202310531299.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2025-11-07
- Estimated Expiration
- 2043-05-12
AI Technical Summary
Existing CNN-based defect detection methods suffer from high false negative rates, inability to achieve global and remote semantic information interaction, and difficulty in aligning input and output images, resulting in low detection accuracy.
A restorative self-supervised defect detection method is adopted, which utilizes a priori masking mechanism and deformable attention mechanism to generate pseudo-anomaly images and anomaly annotation maps. Combined with a priori initial localization module and a defect detection module, the method performs preliminary localization and accurate detection of anomaly areas.
It improves the accuracy and real-time performance of defect detection, generates pseudo-anomaly images that are closer to reality, reduces the number of model parameters, and ensures the accuracy and efficiency of detection.
Smart Images
Figure CN116563250B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application relates to a recovery self-supervised defect detection method and device and a storage medium, and belongs to the technical field of picture detection. BACKGROUND
[0002] Industrial defect detection is one of the most important steps in industrial production, and directly affects product quality and production efficiency. The traditional anomaly detection task is mainly completed manually by skilled workers, which greatly increases the cost of enterprise production. Moreover, manual detection is affected by objective factors such as fatigue and detection environment, resulting in false detection and missed detection, and to some extent, reducing the detection efficiency. With the wide application of artificial intelligence technology in various industries, intelligent defect detection method has become an important means for detecting industrial anomalies. Recently, many researchers have proposed a series of CNN-based defect detection methods, including fully convolutional network, Unet model, deep convolutional autoencoder network model, etc.
[0003] At present, most of the reconstruction anomaly detection algorithms based on CNN are unsupervised. The principle of using unsupervised method for defect detection is that the CNN cannot complete the reconstruction of the abnormal area in the whole detection process. The unsupervised defect detection model only models the distribution of normal data during training, that is, only normal samples are input during the training process, so that the model can learn the distribution of normal samples, and the error between the original image and the reconstructed image is used to realize anomaly detection. For example, the Skip-GaNormaly based on GAN and the MS-FCAE method based on multi-scale feature clustering learn normal samples based on GAN and AE. However, in practice, due to the strong generalization ability of CNN, the abnormality can be well reconstructed, so that the reconstructed image and the defect image cannot well show the difference, resulting in a high false negative rate.
[0004] In summary, defect detection has some defects:
[0005] (1) In unsupervised defect detection, due to the strong generalization ability of CNN, the abnormality can be well reconstructed, so that the difference between the original image and the reconstructed image cannot be well shown, resulting in a high false negative rate;
[0006] (2) The defect detection algorithm based on CNN cannot realize the interaction of global and long-range semantic information, resulting in the inability to generate ideal recovery images;
[0007] (3) It is difficult to keep the input image and the output image aligned, for example, the image style is inconsistent or there is a shift, so when calculating the difference between the output and the input, it is easy to cause false detection at the pixel level. SUMMARY
[0008] The purpose of the present application is to overcome the deficiencies in the prior art, provide a recovery type self-supervised defect detection method, device and storage medium, which can generate more realistic pseudo anomalies, and a prior mask mechanism is proposed, which can pre-judge the defect area for masking, so as to ensure that the model will not learn the information of the defect area during image restoration.
[0009] To achieve the above purpose, the present application is implemented by using the following technical solutions:
[0010] In a first aspect, the present application provides a recovery type self-supervised defect detection method, comprising:
[0011] Obtaining a non-defect image;
[0012] Selecting to add pseudo anomalies or not to add pseudo anomalies; if pseudo anomalies are selected, selecting any continuous area in the non-defect image for masking, introducing a transparency factor to fuse the masked area and the image, or randomly cropping any area for rotation and randomly adjusting the pixels in the area, or selecting both of the two pseudo anomaly generation methods to obtain a generated pseudo anomaly image and an anomaly label image;
[0013] Inputting the pseudo anomaly image into a pre-trained prior initial positioning module, which adopts an anomaly diffusion positioning strategy, divides the image into 32x32 pixel patches, extracts the texture features in each patch to obtain a vector representing the texture features; inputting the vector into the prior initial positioning module, using the prior initial positioning module to determine whether a patch is abnormal, if it is abnormal, masking it, if it is not abnormal, not masking it, so as to achieve the purpose of initially positioning the defect area and masking the defect area, and obtaining a masked image;
[0014] Inputting the masked image into a pre-trained defect detection module to obtain a predicted anomaly label image, first performing maximum and minimum normalization on the anomaly label image and then performing Gaussian smoothing to remove noise in the predicted anomaly label image;
[0015] After obtaining the noise-removed anomaly label image, performing anomaly judgment and positioning according to the anomaly score of the image.
[0016] Further, the training method of the prior initial positioning module comprises:
[0017] Obtaining a training set image, the training set image being a non-defect image;
[0018] Dividing the training set image into 32x32 pixel image blocks, one image being divided into 64 patches, and extracting the texture features in each patch;
[0019] For each pixel in a patch, the gray values of the 8 neighboring pixels are compared with it, if the surrounding pixel values are greater than the center pixel value, the position of the pixel is marked as 1, otherwise 0;
[0020] After comparison in the 3*3 neighborhood, an 8-bit binary number is generated, which represents the center pixel texture value;
[0021] The histogram of each patch is calculated, and then the histogram is normalized to obtain a vector representing the texture feature;
[0022] The vector is input into the prior primary positioning module for training to obtain the trained prior primary positioning module.
[0023] Further, the vector is input into the prior primary positioning module for training to obtain the trained prior primary positioning module, comprising:
[0024] The vector is input into the prior primary positioning module, and a combined kernel K determined by four kernel functions is calculated:
[0025]
[0026] μ q is the weight of the i-th kernel, and ||ω||2=1, K i is the i-th kernel matrix after centering, and there are four kernel matrices in total, the weight μ is a non-zero vector, and the calculation formula is as follows:
[0027]
[0028] Where b=(<K1,yy T > F ,...,<K q ,yy T >F) T , y=(1,1,...,1) T , M ql =<K q , K l > F , q, l∈[1, 4], <.,.> F is the Frobenius inner product, and the obtained μ is introduced into the one-class support vector machine, and the optimization problem is:
[0029]
[0030]
[0031]
[0032] Where, α iis a sample point x i is a corresponding Lagrange multiplier, mu q is a weight of the qth kernel, and K(·, ·) is a kernel function.
[0033] The decision function is as follows:
[0034]
[0035] After obtaining the decision function, the prior initial positioning module is trained to obtain the trained prior initial positioning module.
[0036] Further, the method for generating the pseudo-abnormal picture and the abnormal label picture comprises:
[0037] The size of the input image is transformed into 256x256, the mean and variance are set, and the image is standardized;
[0038] Three variables are defined as flag_1, flag_2 and falg_3, the values of flag_1, falg_2 and falg_3 are 0 or 1, and the variables are randomly assigned values when the input is a flawless picture;
[0039] If flag_1=0, no pseudo-abnormality is generated; if flag_1=1, a pseudo-abnormality is generated;
[0040] If flag_1=1, flag_2=0, and flag_3=0, an arbitrary image region in this category is randomly cropped, the region is flipped and the saturation is adjusted, and the pixel position of the region is randomly adjusted, and placed in an arbitrary region of the flawless picture, to obtain the generated pseudo-abnormal image and abnormal label image;
[0041] If flag_1=1, flag_2=1, and flag_3=0, a 32x32 pixel image region is selected as a center point, and an arbitrary continuous region in the image region is selected to be masked, a transparency factor is introduced to fuse the masked region and the image, to obtain the generated pseudo-abnormal picture and abnormal label picture;
[0042] If flag_1=1 and flag_3=1, two abnormal generation methods are used at the same time to obtain the generated pseudo-abnormal picture and abnormal label picture.
[0043] Further, the training method of the defect detection module comprises:
[0044] The masked picture is input into a defect detection module, the defect detection module is composed of two U-shaped structures, one U-shaped structure includes a deformable transformer model, the transformer model includes an encoder composed of multiple layers of transformers and a decoder composed of multiple layers of transformers and a skip link, wherein each module of the encoder is composed of a deformable attention module and a patch merging layer, and the decoder is composed of a deformable attention module and an upsampling module;
[0045] The input picture is divided into multiple patches while embedding position information, and then input into a deformable attention module for feature learning;
[0046] After passing through the deformable attention module and the patch mering layer of the encoder, the image is upsampled and subjected to deformable attention mechanism operation by the decoder to obtain a restored image, wherein the upsampling operation is a pixel shuffle and bilinear interpolation operation;
[0047] The U-shaped deformable transformer model outputs a restored picture, which is spliced with the generated pseudo abnormal picture and then input into a Unet network for classification to obtain a predicted abnormal annotation picture, the size of the picture is 256*256, maximum and minimum normalization and Gaussian smoothing operations are performed on the abnormal annotation picture to remove noise, and a binaryzation operation is performed on the picture to obtain a binaryzation picture M, wherein 0 indicates that the pixel is normal and 1 indicates that the pixel is abnormal;
[0048] The L2 loss and the SSIM loss of the input picture X and the restored picture I are calculated, and the loss is denoted as the restoration loss loss inpaint :
[0049] loss inpaint = SSIM(X, I) + L2(X, I)
[0050] The focal loss of the abnormal annotation picture A and the binaryzation abnormal annotation picture M is calculated, and the loss is denoted as the discrimination loss loss judge :
[0051] loss judge = -α t (1-p t ) γ log(p t )
[0052] Wherein, α t is an adjustment factor, p t is a probability estimate of predicting a normal pixel or an abnormal pixel, and the total loss loss totalThe sum of the two, where α is the penalty coefficient:
[0053] loss total =loss inpaint +αloss judge
[0054] The model is iterated backward using the stochastic gradient descent algorithm to update the weights, obtain the optimal loss, and thus obtain the optimal model. The parameters with the optimal loss are then saved.
[0055] Furthermore, the calculation method for the deformable attention mechanism is as follows:
[0056] Let Q, K, and V be the query, key, and value matrices, respectively. Let Q... i The j-th token is Q ij The decision network is based on Q. ij Generate window size ij and offset ij :
[0057] size ij offset ij =Q ij W iD
[0058] s ij =σ1(size ij )×L,o ij =σ2(offset) ij )×L
[0059] Where j∈[0,L-1] represents the sequence dimension of the patch, W iD It is a parameter matrix; σ1 and σ2 are two nonlinear functions used to limit the output range, and the sigmoid function needs to be applied to adjust the size. ij Limit it to the range (0, 1), and then scale it using L;
[0060] After obtaining the window size and offset, an anchor point A can be obtained. ij Combined with window size s ij and offset o ij This allows us to obtain the left edge of the attention window. ij and right boundary ij :
[0061] A ij =j+o ij
[0062] Left ij =A ij -s ijRight ij = A ij + s ij
[0063] wherein j is the current position index, and the final deformable attention mechanism calculation formula is as follows:
[0064]
[0065] deformable(Q, K, V) = concat(head1, …, head h )W o .
[0066] Further, the test method of the defect detection module comprises:
[0067] sending a test picture into the trained prior initial positioning module, dividing the training picture into 32*32 pixel image blocks, processing the image blocks to obtain 64 texture vectors, sending the texture vectors into the prior initial positioning module to obtain a detection result, marking a defect-free patch as 0 and a defect patch as 1, calculating a loss, saving an optimal model, obtaining a detection result, marking a defect-free patch as 0 and a defect patch as 1, and performing masking on the patch marked as 1;
[0068] sending the masked picture into the trained defect detection module to obtain a predicted abnormality annotation map, the size of the map being 256*256, performing maximum and minimum normalization operation and Gaussian smoothing on the abnormality annotation map to remove noise, and finally the pixel-level abnormality score being represented by taking the maximum value of the smoothed abnormality annotation map.
[0069] In a second aspect, the present application provides a recovery type self-supervised defect detection device, comprising:
[0070] a picture acquisition module configured to acquire a defect-free picture;
[0071] an abnormality map generation module configured to select adding a pseudo abnormality or not adding a pseudo abnormality; if the pseudo abnormality is selected, a mask is performed on an arbitrary continuous region in the defect-free picture, a transparency factor is introduced to fuse the masked region and the image, or an arbitrary region is randomly cropped, rotated and randomly adjusted, or both of the two pseudo abnormality generation methods are selected, so as to obtain a generated pseudo abnormality picture and an abnormality annotation map;
[0072] A preliminary positioning module is used for inputting the pseudo-abnormal picture into a pre-trained priori preliminary positioning module, dividing the picture into 32*32 pixel patches, extracting texture features in each patch to obtain a vector representing the texture features, inputting the vector into the priori preliminary positioning module, using the priori preliminary positioning module to determine whether a patch is abnormal, and if it is abnormal, masking it, and if it is not abnormal, not masking it, so as to preliminarily locate the defect area and mask the defect area, and obtain a masked picture.
[0073] A detection module is used for inputting the masked picture into a pre-trained defect detection module to obtain a predicted abnormal annotation map, performing maximum and minimum normalization on the abnormal annotation map, and then performing Gaussian smoothing to remove noise in the predicted abnormal annotation map.
[0074] An abnormality judgment and positioning module is used for performing abnormality judgment and positioning according to the abnormal score of the abnormal annotation map after removing the noise.
[0075] In a third aspect, the present application provides an electronic device comprising a processor and a storage medium.
[0076] The storage medium is used for storing instructions.
[0077] The processor is used for operating according to the instructions to perform the steps of the method according to any one of the preceding aspects.
[0078] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the steps of the method according to any one of the preceding aspects.
[0079] Compared with the prior art, the present application has the following beneficial effects:
[0080] The present application provides a restoration type self-supervised defect detection method, device and storage medium, uses a pseudo-abnormal generation method based on a random selection and transparency selection mechanism to generate a defect picture, uses a pre-trained priori module to preliminarily locate a defect area, masks the located defect part, inputs it into a defect detection module for training to obtain a detection result and an abnormal positioning map. This method can generate more realistic pseudo-abnormalities, and a priori masking mechanism is proposed, which uses an abnormality diffusion positioning strategy to pre-judge the defect area for masking, so as to ensure that the model does not learn the information of the defect area during image restoration, thereby improving the accuracy of the model. The defect detection module uses a deformable attention mechanism, which reduces the model parameter amount and ensures the real-time performance of the model. BRIEF DESCRIPTION OF DRAWINGS
[0081] Figure 1 is a flowchart of a restoration type self-supervised defect detection method provided by an embodiment of the present application.
[0082] Figure 2 is a pre-mask flowchart provided by the embodiment of the application. DETAILED DESCRIPTION
[0083] The application will be further described below with reference to the drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the application, and cannot be used to limit the protection scope of the application.
[0084] Embodiment 1
[0085] This embodiment introduces a recovery type self-supervised defect detection method, which comprises:
[0086] Obtain a defect-free picture;
[0087] Select to add pseudo-exceptions or not to add pseudo-exceptions; if pseudo-exceptions are selected, select any continuous area in the defect-free picture to mask, introduce a transparency factor to fuse the masked area and the image, or randomly crop any area to rotate and randomly adjust the pixels in the area, or select both of the two pseudo-exception generation methods at the same time, so as to obtain a generated pseudo-exception picture and an exception annotation map;
[0088] Input the pseudo-exception picture into a pre-trained prior initial positioning module, which adopts an exception diffusion positioning strategy, divides the picture into 32x32 pixel patches, extracts the texture features in each patch to obtain a vector representing the texture features; input the vector into the prior initial positioning module, use the prior initial positioning module to determine whether a patch is abnormal, if it is abnormal, mask it, if it is not abnormal, do not mask it, so as to achieve the purpose of initially positioning the defect area and masking the defect area, and obtain a masked picture;
[0089] Input the masked picture into a pre-trained defect detection module to obtain a predicted exception annotation map, and perform maximum and minimum normalization and then Gaussian smoothing on the exception annotation map to remove noise in the predicted exception annotation map;
[0090] After obtaining the noise-removed exception annotation map, perform exception judgment and positioning according to the exception score of the map.
[0091] As shown in Figure 1 The recovery type self-supervised defect detection method provided by the embodiment, and the application process thereof specifically involves the following steps:
[0092] Step 1: image preprocessing
[0093] 1.1 Experiments are performed using DAGM dataset. DAGM dataset is a well-known benchmark dataset for surface defect detection. It contains ten gray scale images of different computer generated surfaces and various defects, such as scratches or spots. Each dataset consists of "defect-free" and "defect" pictures.
[0094] 1.2 Transform the input image size to 256x256, set the mean and variance, and normalize the image.
[0095] Step 2: Train the prior pre-localization module
[0096] 2.1 Read in the dataset processed in step 1.2; first input the training set pictures (training set pictures are all defect-free pictures), divide the input picture into 32x32 pixel image blocks, one picture is divided into 64 patches, and extract the texture features in each patch.
[0097] 2.2 For each pixel in the patch, compare the gray scale values of the adjacent 8 pixels with it, if the surrounding pixel value is greater than the center pixel value, the position of the pixel is marked as 1, otherwise as 0. After comparison in the 3x3 neighborhood, an 8-bit binary number is generated, which can represent the texture value of the center pixel.
[0098] 2.3 Calculate the histogram of each patch, then normalize the histogram to obtain the vector representing the texture feature.
[0099] 2.4 Input the vector into MK-OCSVM, MK-OCSVM is a single-class support vector machine based on linear kernel function, Gaussian kernel function, Laplacian kernel function and polynomial kernel function, first calculate the combined kernel K determined by the four kernel functions:
[0100]
[0101] μ q is the weight of the i-th kernel, and || μ ||2=1, K i is the i-th kernel matrix after centering, and there are four kernel matrices. The weight μ is a non-zero vector, and the calculation formula is as follows:
[0102]
[0103] where b = (<K1, y T > F ,..., <K q , y T > F T , y = (1, 1,..., 1) T , Mql = <K q , K l > = <K F , q, l ∈ [1, 4], <.,.> F is the Frobenius inner product. Introducing μ into the one-class support vector machine, the optimization problem is:
[0104]
[0105]
[0106]
[0107] where α i is the Lagrange multiplier corresponding to the sample point x i , μ q is the weight of the qth kernel, and K(·, ·) is the kernel function.
[0108] The decision function is as follows:
[0109]
[0110] 2.5 After obtaining the decision function, the MK-OCSVM is trained to obtain the trained MK-OCSVM.
[0111] Step 3: Generating simulated anomalies
[0112] 3.1 Read in the training set processed in step 1.2, and define three variables flag_1, flag_2 and falg_3, the values of flag_1, falg_2 and falg_3 are 0 or 1, and randomly assign values to the variables when inputting the flawless picture;
[0113] 3.2 If flag_1 = 0, no pseudo anomaly is generated; if flag_1 = 1, a pseudo anomaly is generated;
[0114] 3.3 If flag_1 = 1, flag_2 = 0, and flag_3 = 0, randomly crop an image region in this category, flip and adjust the saturation of the region, and randomly adjust the pixel position of the region, and place it in any region of the flawless picture to obtain the generated pseudo anomaly image and anomaly label map;
[0115] 3.3 If flag_1 = 1, flag_2 = 1, and flag_3 = 0, randomly select a point in the image as the center point to select a 32x32 pixel image region, select any continuous region in the image region to mask, and introduce a transparency factor to fuse the masked region and the image to obtain the generated pseudo anomaly picture and anomaly label map.
[0116] 3.4 If flag_1 = 1 and flag_3 = 1, then both exception generation methods are used simultaneously to obtain the generated pseudo-exception image and exception annotation image;
[0117] Step 4: Preliminary assessment of defect areas
[0118] The pseudo-anomaly image generated in step 3 is fed into the trained prior localization module. The prior localization module adopts an anomaly diffusion localization strategy. This module divides the image into blocks, judges the image blocks, and masks the anomaly blocks to amplify the defect localization area and avoid missing the anomaly area. The training image is divided into 32×32 pixel image blocks. The operations in steps 2.2 and 2.3 are repeated to obtain 64 texture vectors. The texture vectors are fed into the prior localization module to obtain the detection results. The prior localization module is a single classifier determined by four kernel functions. Patches without defects are marked as 0, and patches with defects are marked as 1. Patches with defects are masked.
[0119] Step 5: Train the defect detection module
[0120] 5.1 The masked image obtained in step 4 is fed into the defect detection module. The defect detection model consists of two U-shaped structures. The image is first fed into a U-shaped deformable transformer model, which consists of an encoder composed of multiple transformer layers, a decoder composed of multiple transformer layers, and skip connections. Each module of the encoder consists of a deformable attention module and a patch merging layer, and the decoder consists of a deformable attention module and an upsampling module.
[0121] 5.2 The input image is divided into multiple patches, and positional information is embedded within each patch. These patches then enter the deformable attention module for feature learning. This module uses a simple decision network to change the size and position of the multi-head attention window, thereby reducing the number of parameters and learning local features.
[0122] 5.3 Let Q, K, and V be the query, key, and value matrices, respectively. Let Q... i The j-th token is Q ij The decision network is based on Q. ij Generate window size ij and offset ij :
[0123] size ij offset ij =Q ij W iD
[0124] s ij = σ1(size ij ) x L, o ij = σ2(offset ij ) x L
[0125] where j ∈ [0, L-1] represents the sequence dimension of the patch, W iD is the parameter matrix; σ1 and σ2 are two nonlinear functions to limit the output range. The sigmoid function needs to be applied to limit the size ij between (0, 1), and then scaled by L.
[0126] After obtaining the window size and offset, an anchor point A ij can be obtained, and in combination with the window size s ij and the offset o ij , the left boundary Left ij and the right boundary Right ij of the attention window can be obtained:
[0127] A ij = j + o ij
[0128] Left ij = A ij - s ij , Right ij = A ij + s ij
[0129] where j is the current position index, and the final deformable attention mechanism calculation formula is as follows:
[0130]
[0131] deformable(Q, K, V) = concat(head1, …, head h )W o
[0132] 5.4 After the multi-layer deformable attention mechanism of the encoder and the patch mering, the decoder is used for upsampling and deformable attention mechanism operation to obtain the restored image, wherein the upsampling operation is the PixelShuffle and bilinear interpolation operation.
[0133] The U-shaped deformable transformer model outputs a restored picture, which is input into the Unet network for discrimination to obtain a predicted abnormality annotation picture. The size of the picture is 256x256. The abnormality annotation picture is subjected to maximum and minimum normalization and Gaussian smoothing operation to remove noise. The picture is further binarized to obtain a binarized abnormality annotation picture M.
[0134] Step 6: Calculate the loss
[0135] 6.1 Calculate the L2 loss and SSIM loss of the input picture X and the restored picture I, which is denoted as the restoration loss loss inpaint :
[0136] loss inpaint = SSIM(X, I) + L2(X, I)
[0137] Calculate the focal loss of the abnormality annotation picture A and the predicted binarized abnormality annotation picture M, denoted as the discrimination loss loss judge :
[0138] loss judge = -a t (1-p t ) γ log(p t )
[0139] Where a t is an adjustment factor, and p t is a probability estimate for predicting normal pixels or abnormal pixels. The total loss loss total is the sum of the two, and a is a penalty coefficient:
[0140] loss total = loss inpaint + a loss judge
[0141] 6.2 Use the stochastic gradient descent algorithm to iteratively update the weights of the model to obtain the optimal loss, thereby obtaining the optimal model, and save the parameters with the optimal loss.
[0142] Step 7: Test the model
[0143] 7.1 Input the test picture into the trained MK-OCSVM, divide the training picture into 32x32 pixel image blocks, and repeat the operations of 2.2 and 2.3 to obtain 64 texture vectors. Input the texture vectors into the prior module to obtain the detection result. The patch without defects is marked as 0, and the patch with defects is marked as 1. Calculate the loss and save the optimal model. The detection result is obtained. The patch without defects is marked as 0, and the patch with defects is marked as 1. The patch marked as 1 is masked.
[0144] 7.2 The masked picture is input into the trained defect detection network to obtain a predicted abnormality annotation map with a size of 256x256. The abnormality annotation map is subjected to maximum-minimum normalization and Gaussian smoothing to remove noise. The final pixel-level abnormality score is represented by taking the maximum value of the smoothed abnormality annotation map.
[0145] Embodiment 2
[0146] The embodiment provides a recovery type self-supervised defect detection device, which comprises:
[0147] a picture acquisition module configured to acquire a defect-free picture;
[0148] an abnormality map generation module configured to select to add a pseudo abnormality or not to add the pseudo abnormality. If the pseudo abnormality is selected to be added, a continuous region in the defect-free picture is selected to be masked, a transparency factor is introduced to fuse the masked region and the picture, or an arbitrary region is randomly cropped, rotated and subjected to random adjustment of pixels in the region, or both of the two pseudo abnormality generation methods are selected to obtain a generated pseudo abnormality picture and an abnormality annotation map;
[0149] a preliminary positioning module configured to input the pseudo abnormality picture into a pre-trained prior preliminary positioning module, divide the picture into 32x32 pixel patches, extract texture features in each patch to obtain a vector representing the texture features, input the vector into the prior preliminary positioning module, and use the prior preliminary positioning module to determine whether a patch is abnormal. If the patch is abnormal, the patch is masked. If the patch is not abnormal, the patch is not masked, so as to preliminarily position a defect region and mask the defect region to obtain a masked picture;
[0150] a detection module configured to input the masked picture into a pre-trained defect detection module to obtain a predicted abnormality annotation map, and perform maximum-minimum normalization and Gaussian smoothing on the abnormality annotation map to remove noise in the predicted abnormality annotation map;
[0151] an abnormality judgment and positioning module configured to, after obtaining the abnormality annotation map from which the noise is removed, perform abnormality judgment and positioning according to an abnormality score of the abnormality annotation map.
[0152] Embodiment 3
[0153] The embodiment provides a computer-readable storage medium having a computer program stored thereon. The program is executed by a processor to implement the steps of the method in any one of the embodiments 1.
[0154] The above merely describes the preferred embodiments of the present application, and it should be pointed out that, for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.
Claims
1. A method of defect detection with a self-supervised approach, characterized in that, The method comprises the following steps: Obtain a non-defect image; Select to add pseudo-defects or not to add pseudo-defects; If it is selected to add pseudo-defects, select an arbitrary continuous region in the non-defect image to mask, introduce a transparency factor to fuse the masked region and the image, or randomly crop an arbitrary region to rotate and randomly adjust the pixels in the region, or select both of the two pseudo-defect generation methods to obtain a generated pseudo-defect image and an abnormality annotation map; Input the pseudo-defect image into a pre-trained prior initial positioning module, the module adopts an abnormality diffusion positioning strategy, divides the image into 32*32 pixel patches, extracts the texture features in each patch to obtain a vector representing the texture features, inputs the vector into the prior initial positioning module, and uses the prior initial positioning module to determine whether a patch is abnormal, and if the patch is abnormal, masks the patch, and if the patch is not abnormal, does not mask the patch, so as to preliminarily position the defect region and mask the defect region, and obtain a masked image; Input the masked image into a pre-trained defect detection module to obtain a predicted abnormality annotation map, and perform maximum-minimum normalization and Gaussian smoothing on the abnormality annotation map to remove noise in the predicted abnormality annotation map; After obtaining the abnormality annotation map without noise, perform abnormality judgment and positioning according to the abnormality score of the map.
2. The method of claim 1, wherein the method further comprises: The training method of the prior initial positioning module comprises the following steps: Obtain a training set image, wherein the training set image is a non-defect image; Divide the training set image into 32*32 pixel image blocks, and divide one training set image into 64 patches, and extract the texture features in each patch; For the pixels in each patch, compare the gray values of the adjacent 8 pixels with the center pixel, and if the values of the surrounding pixels are greater than the value of the center pixel, the position of the pixel point is marked as 1, otherwise as 0; After comparison in a 3*3 neighborhood, an 8-bit binary number is generated, which represents the texture value of the center pixel; Calculate the histogram of each patch, and then normalize the histogram to obtain a vector representing the texture features; Input the vector into the prior initial positioning module for training to obtain the trained prior initial positioning module.
3. The method of claim 2, wherein the method further comprises: The training method of the prior initial positioning module comprises the following steps: Input the vector into the prior initial positioning module, and calculate the combined kernel K determined by the four kernel functions: ； is the weight of the ith core, and , is the centralized ith basis core matrix, and there are four basis core matrices in total, and the weights is a non-zero vector, and the calculation formula is as follows: ； where , , , , is the Frobenius inner product, and Introducing the one-class SVM, the optimization problem becomes: ； ； ； wherein, is a sample point corresponding Lagrange multiplier, is a weight of the qth kernel, is a kernel function; The decision function is as follows: ； After obtaining the decision function, train the prior initial positioning module to obtain the trained prior initial positioning module.
4. The method of claim 1, wherein the method further comprises: The method for generating the pseudo-defect image and the abnormality annotation map comprises the following steps: Transform the size of the input image to 256*256 pixels, set the mean and variance, and standardize the image; Define three variables as flag_1, flag_2 and falg_3, the values of flag_1, falg_2 and falg_3 are 0 or 1, and the variables are randomly assigned values when the non-defect image is input; If flag_1=0, no pseudo-defect is generated; if flag_1=1, a pseudo-defect is generated; If flag_1=1, flag_2=0, flag_3=0, randomly crop an image region in the category, flip and adjust the saturation of the region, and randomly adjust the pixel position of the region, and place it in any area of the defect-free picture to obtain a generated pseudo- abnormal image and an abnormal label image; If flag_1=1, flag_2=1, flag_3=0, randomly select a 32x32 pixel image region with a center point, and select any continuous region in the image region to mask, introduce a transparency factor to fuse the masked region and the image, and obtain a generated pseudo- abnormal picture and an abnormal label image; If flag_1=1, flag_3=1, two kinds of abnormal generation methods are used at the same time to obtain a generated pseudo- abnormal picture and an abnormal label image.
5. The method of claim 1, wherein the method further comprises: The training method of the defect detection module comprises: The masked picture is input into the defect detection module, which is composed of two U-shaped structures, and one U-shaped structure includes a deformable transformer model, the transformer model includes an encoder composed of multiple layers of transformer and a decoder composed of multiple layers of transformer and a jump link, wherein each module of the encoder is composed of a deformable attention module and a Patch Merging layer, and the decoder is composed of a deformable attention module and an up-sampling module; The input picture is divided into multiple patches, and the position information is embedded at the same time, and then enters the deformable attention module for feature learning; After passing through the deformable attention module and the Patch mering layer of the encoder, the decoder is used for up-sampling and deformable attention mechanism operation to obtain a restored image, wherein the up-sampling operation is PixelShuffle and bilinear interpolation operation; The U-shaped deformable transformer model outputs a restored picture, which is spliced with the generated pseudo- abnormal picture and then sent to the Unet network for classification to obtain a predicted abnormal label image, and the size of the image is 256x256, the maximum and minimum value normalization and Gaussian smoothing operation are performed on the abnormal label image to remove noise, and the binary operation is performed on the picture to obtain a binary picture M, wherein 0 represents that the pixel is normal, and 1 represents that it is abnormal; The L2 loss and SSIM loss of the input picture X and the recovered picture Y are calculated, and the loss is denoted as the recovery loss Lr : Lr = L2(X, Y) + SSIM(X, Y) : ； The focal loss of the abnormal annotation graph A and the binarized abnormal annotation graph M is calculated, denoted as the discriminant loss : ； wherein is a tuning factor, is a probability estimate that a pixel is predicted to be normal or abnormal, total loss is the sum of both, is a penalty coefficient: ； The random gradient descent algorithm is used to iteratively update the weights of the model to obtain the optimal loss and thus the optimal model, and the parameters with the optimal loss are saved.
6. The method of claim 5, wherein the method further comprises: The deformable attention mechanism calculation method is as follows: Let Q, K, and V be the query, key, and value matrices, respectively. The Middle Each token is Decision network based on Generate window size and offset : ； where denotes the sequence dimension of patch, is a parameter matrix; and are two non-linear functions to limit the output range, sigmoid function needs to be applied to limit between (0, 1) and scaled by L; After obtaining the window size and the offset, an anchor point can be obtained , in combination with the window size and the offset , the left boundary and the right boundary of the attention window can be obtained: ； Wherein, j is the current position index, and the final deformable attention mechanism calculation formula is as follows: 。 7. The method of claim 5, wherein the method further comprises: The test method of the defect detection module comprises: The test picture is sent into the trained prior initial positioning module, the training picture is divided into 32*32 pixel image blocks, the image blocks are processed, 64 texture vectors are obtained, the texture vectors are sent into the prior initial positioning module, the detection result is obtained, the patch without defect is marked as 0, the patch with defect is marked as 1, the loss is calculated, the optimal model is saved, and the detection result is obtained, the patch without defect is marked as 0, the patch with defect is marked as 1, and the patch marked as 1 is masked; The masked picture is sent into the trained defect detection module to obtain a predicted abnormality annotation map, the size of the map is 256*256 pixels, the maximum and minimum normalization operation and the Gaussian smoothing are performed on the abnormality annotation map to remove noise, and finally the abnormality score at the pixel level is represented by taking the maximum value of the smoothed abnormality annotation map.
8. A device for defect detection with self-supervision by restoration, characterized in that Comprise: A picture acquisition module is configured to acquire a defect-free picture. An abnormality map generation module is configured to select to add pseudo-abnormalities or not to add pseudo-abnormalities. If it is selected to add pseudo-abnormalities, any continuous region in the defect-free picture is selected to be masked, a transparency factor is introduced to fuse the masked region and the image, or any region is randomly cropped, rotated and randomly adjusted, or both of the two pseudo-abnormality generation methods are selected to obtain a generated pseudo-abnormality picture and an abnormality annotation map. A preliminary positioning module is configured to input the pseudo-abnormality picture into a pre-trained prior initial positioning module, divide the picture into 32*32 pixel patches, extract texture features in each patch to obtain a vector representing the texture features, input the vector into the prior initial positioning module, and use the prior initial positioning module to determine whether a patch is abnormal, if yes, mask the patch, and if not, do not mask the patch, so as to preliminarily locate the defect region and mask the defect region, and obtain a masked picture. A detection module is configured to send the masked picture into a pre-trained defect detection module to obtain a predicted abnormality annotation map, perform maximum and minimum normalization on the abnormality annotation map, and then perform Gaussian smoothing to remove noise in the predicted abnormality annotation map. An abnormality judgment and positioning module is configured to perform abnormality judgment and positioning according to the abnormality score of the abnormality annotation map after removing noise.
9. An electronic device, comprising: Comprise a processor and a storage medium; The storage medium is configured to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1-7.
10. A computer readable storage medium having stored thereon a computer program, characterized in that: The program is executed by the processor to implement the steps of the method according to any one of claims 1-7.
Citation Information
Patent Citations
Knowledge distillation-based unsupervised industrial anomaly detection method and system
CN119991555A