Texture surface defect detection method based on self-attention adversarial autoencoder

Abstract

The present invention discloses a method for detecting surface defects of textures based on an adversarial autoencoder based on self-attention, comprising the following steps: Step 1: According to defect-free texture industrial product images, two defect generation methods based on color and deformation are constructed to generate defect data to simulate real defect samples; Step 2: Build a framework for detecting surface defects of textures based on an adversarial autoencoder based on self-attention; Step 3: Design the metric loss of the input image and the reconstructed image; Step 4: Obtain the abnormal score at the pixel level by calculating the multi-scale structural similarity of the test image and the reconstructed image, and then determine whether the test image has defects. The present invention can accurately locate defects on the surface of a product given a large number of defect-free texture product surface samples; It has high operating efficiency and can meet real-time requirements; And the hardware cost is low, and it is easy to promote in industrial defect detection systems, and has good application prospects.

Description

Technical Field

[0001] The present invention relates to a method for processing images of surface defects of textured industrial products, and in particular to a method for detecting surface defects of industrial products based on a self-attention adversarial autoencoder algorithm. Background Art

[0002] Texture surface defect detection is a key step in industrial production. Its purpose is to identify and locate defects on the product surface (such as Figure 1 ) to determine whether the product is qualified, and then adjust the production plan in time, improve production efficiency and reduce production costs. Traditionally, surface defect detection is performed by human eyes, but human visual evaluation is not only time-consuming and labor-intensive, but also highly subjective and prone to errors. With the reduction of hardware equipment costs and the increase of computing power, industrial product surface defect detection technology based on computer vision has made great progress and has been widely used in industrial production fault detection systems.

[0003] Traditional machine vision defect detection methods, such as support vector machine (SVM) methods, HOG and Gabor feature extraction methods, are used to distinguish defects from non-defects. This feature is not robust enough in the case of complex texture types and scene changes. Therefore, it is difficult to apply it to the process of industrial product surface defect detection. In recent years, autoencoder networks have achieved good performance in tasks such as image reconstruction and defect detection. However, defect detection algorithms based on autoencoder networks still have many problems, such as poor defect localization ability and blurred edges of reconstructed images. Summary of the invention

[0004] In view of the shortcomings of traditional methods, the purpose of the present invention is to provide a self-attention-based adversarial autoencoder texture surface defect detection algorithm, which uses the self-attention-based adversarial autoencoder algorithm to detect defect targets in the acquired texture industrial product surface image. To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0005] Step 1: Based on the defect-free texture industrial product images, two defect generation methods based on color and deformation are built to generate defect data to simulate real defect samples;

[0006] Step 2: Build a self-attention-based adversarial autoencoder texture surface defect detection framework;

[0007] Step 3: Design the metric loss of input image and reconstructed image;

[0008] Step 4: Obtain the pixel-level anomaly score by calculating the multi-scale structural similarity between the test image and the reconstructed image, and then determine whether the test image has defects.

[0009] A further technical solution is that the specific construction steps for generating defect data based on the color method described in step 1 are as follows:

[0010] Step 1-1: Use Bezier curve to create a closed area;

[0011] Step 1-2: Add random colors to the enclosed area;

[0012] Step 1-3: Adjust the enclosed area to a random size;

[0013] Step 1-4: Add closed areas to random locations of the defect-free image to complete the generation of defect data based on the color method.

[0014] A further technical solution is that the specific construction steps for generating defect data based on the deformation method described in step 1 are as follows:

[0015] Step 1-5: Randomly generate two points (p1, p2) and deformation radius r;

[0016] Step 1-6: Calculate the deformation area based on p1, p2 and r;

[0017] Step 1-7: According to the formula Calculate the pixel position of the deformed area and complete the generation of defect data based on the deformation method.

[0018] A further technical solution is that the steps of building the self-attention-based adversarial autoencoder texture surface defect detection framework described in step 2 are as follows:

[0019] Step 2-1: Build the encoder network framework;

[0020] Step 2-2: Build the decoder network framework;

[0021] Step 2-3: Build a self-attention framework that combines channels and positions;

[0022] Step 2-4: Use the discriminator in the DCGAN network to build the discriminator network framework;

[0023] Step 2-5: Use four fully connected layer operations to build the middle layer network framework;

[0024] Step 2-6: Use three fully connected layer operations to build the classifier network framework.

[0025] A further technical solution is that the specific steps of building the encoder network framework described in step 2-1 are as follows:

[0026] Step 2-1-1: Use the first five convolutional modules of the VGG-16 network as the encoding layer network;

[0027] Step 2-1-2: After each convolution module, a ReLUs activation operation with a slope of 0.2 is performed, followed by a maximum pooling operation with a kernel size of 2*2 and a stride of 2.

[0028] A further technical solution is that the specific operation steps of building the decoder network framework described in step 2-2 are as follows:

[0029] Step 2-2-1: For each convolution block in the encoder, there is a corresponding decoding block in the decoder. The first four decoding blocks contain a convolution layer and a deconvolution layer. The convolution layer uses a 3*3 convolution kernel, a step size of 1, and a padding of 1. The deconvolution layer uses a 4*4 convolution kernel, a step size of 2, and a padding of 1. After the convolution layer and the deconvolution layer, a ReLUs activation operation with a slope of 0.2 is used;

[0030] Step 2-2-2: The fifth decoding block contains a convolution layer with a convolution kernel size of 3*3, a stride of 1, and a padding of 1.

[0031] A further technical solution is that the specific steps of building the channel self-attention framework described in step 2-3 are as follows:

[0032] Step 2-3-1: Output the features of the last layer of the encoder module Input to the channel self-attention module;

[0033] Step 2-3-2: Re-adjust M A The shape is M A1 , where C represents the number of channels, N = H × W represents M A The number of features in

[0034] Step 2-3-3: Through M A1 With M A1 The channel attention map is obtained by multiplying the transposed matrix of

[0035] Step 2-3-4: Then apply the softmax operation to obtain the final channel attention map X;

[0036] Step 2-3-5: Use Formula M out =[X·M A1 ] R +M A Apply the channel self-attention map to the feature map, [·] R is an operation that reshapes the matrix.

[0037] A further technical solution is that the specific operation steps for building the position self-attention framework described in step 2-3 are as follows:

[0038] Step 2-3-6: Output M of the channel self-attention module B The features of are input into three convolutional layers to generate three new feature maps M respectively. B1 , M B2 , and M B3 ,in Reshape to Where C represents the number of channels, N = H × W represents M B The number of features in

[0039] Step 2-3-7: Calculate M B2 and M B1 The cosine distance between the transposes of , and the position self-attention map is obtained;

[0040] Step 2-3-8: Use softmax operation to obtain the final position self-attention map S;

[0041] Step 2-3-9: Use Formula M out =[M B3 ·S] R +M B Apply the position self-attention map to the feature map.

[0042] A further technical solution is that the design of the metric loss of the input image and the reconstructed image described in step 3 is specifically described as follows:

[0043] Step 3-1: Considering the structural similarity between the reconstructed image and the input image and the attention map F of the input image, the reconstruction loss is designed to be L re (i,j)=SSIM(I in (i,j),I re (i,j))*(1+C r *F(i,j)), where L re (i,j) represents the loss of the i,jth position of the image, I in represents the input image, I re represents the reconstructed image, SSIM(·) represents the structural similarity loss, C r ∈{0,1} represents the classification result, and F(·) represents the attention map of the specified convolutional layer;

[0044] Step 3-2: Propose edge consistency loss in gradient domain Make the edges of the input image consistent with the reconstructed image. Here, λ edge is a constant, U is the number of downsampling times, ↓u represents the downsampling operation, ||·|| F represents the F norm, σ() represents the weakly corrected linear unit, represents the gradient operation, ⊙ represents the corresponding element multiplication operation;

[0045] Step 3-3: L = θ1L re +θ2L edge +θ3L cls +θ4L d , where θi is a constant, L d To combat the loss, L cls is the cross entropy classification loss.

[0046]

[0047] L re =SSIM(I in , I re )*(1+C r *F))

[0048]

[0049] A further technical solution is that the determination of whether the test image described in step 4 is defective is specifically described as follows:

[0050] Step 4-1: Use multi-scale structural similarity to calculate the anomaly map ASM (I o ,I r ); Here, I o represents the test image, I r represents the reconstructed image,

[0051] Step 4-2: According to the abnormal image, by setting the segmentation threshold τ1 and the classification threshold τ2, the final pixel-level defect segmentation map and image-level defect classification category are obtained.

[0052] Considering the shortcomings of the traditional autoencoder network in its poor defect localization ability, the present invention introduces a self-attention mechanism in the encoding layer so that long-range context information can be adaptively aggregated to improve the feature expression of texture reconstruction, such as Figure 2 As shown. In order to solve the problem of poor robustness of the reconstruction results of defective areas, the present invention produces defect samples from the two aspects of color and deformation, so that the model can accurately classify the produced defect samples. In addition, in the reconstruction process, the two input and output modes (defect-no defect, no defect-no defect) make the reconstruction results more robust. In order to solve the problem of blurred edges of the reconstruction results, the present invention adds edge consistency loss so that the input image and the reconstructed image are consistent in terms of edges to eliminate blur.

[0053] Compared with the prior art, the present invention has the following significant advantages: (1) unsupervised defect detection of textured industrial product surface images without the need to provide defect samples or labels; (2) high accuracy of industrial product surface defect detection; (3) fast operation speed, able to meet the real-time requirements of practical applications; (4) simple hardware requirements, easy to promote on a large scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 These are normal images and abnormal images of the textured product of the present invention; the first row is the normal image, the second row is the abnormal image, and the third row is a partial magnified image of the abnormal image.

[0055] Figure 2 It is a network framework diagram of the present invention.

[0056] Figure 3 It is a schematic diagram of the defect making process based on the deformation method of the present invention.

[0057] Figure 4 This is a result diagram of the method for making two defects based on color and deformation according to the present invention.

[0058] Figure 5 It is the defect reconstruction and detection result diagram of the present invention, and the first to fourth rows respectively represent the test image, the reconstructed image, the defect location result diagram and the true value diagram. DETAILED DESCRIPTION

[0059] like Figure 1 As shown, the steps of the self-attention-based adversarial autoencoder texture surface defect detection algorithm of the present invention are as follows:

[0060] Step 1: Based on the defect-free texture industrial product images, two defect generation methods based on color and deformation are built to generate defect data to simulate real defect samples;

[0061] Step 2: Build a self-attention-based adversarial autoencoder texture surface defect detection framework;

[0062] Step 3: Design the metric loss of input image and reconstructed image;

[0063] Step 4: Obtain the pixel-level anomaly score by calculating the multi-scale structural similarity between the test image and the reconstructed image, and then determine whether the test image has defects.

[0064] The present invention discloses a method for detecting surface defects of an adversarial autoencoder texture based on self-attention, and proposes an unsupervised solution for abnormal detection of the surface of industrial products. The method combines the autoencoder with the self-attention mechanism to detect defects on the surface of textured industrial products. First, defect data is generated based on color and deformation to simulate real defect samples. Secondly, two input and output modes (no defect-no defect, defect-no defect) are used to train the adversarial autoencoder defect detection algorithm based on self-attention. Finally, the pixel-level abnormality score is obtained by calculating the multi-scale structural similarity of the test image and the reconstructed image, and then it is judged whether the test image has defects. The present invention can accurately locate the defects on the surface of the product given a large number of defect-free texture product surface samples; the operation efficiency is high, and the real-time requirements can be met; and the hardware cost is low, and it is easy to promote in the industrial defect detection system, and has good application prospects.

[0065] The specific implementation is as follows.

[0066] Step 1: Generate defect data based on color and deformation methods.

[0067] According to the given normal texture surface sample (such as Figure 1 As shown in the first row), the present invention generates defect data based on two defect generation methods, color and deformation, to simulate real defect samples (such as Figure 1 shown in the second row).

[0068] The generation of color defect samples is mainly divided into the following steps. First, a closed region is established using the Bezier curve. Second, random colors are added to the closed region. Third, the closed region is adjusted to a random size. Finally, the closed region is added to a random position of the defect-free image.

[0069] The production of deformation defect samples is mainly divided into the following steps. First, two points (p1, p2) and deformation radius r are randomly generated. Second, the deformation area is calculated based on p1, p2 and r. Finally, according to the formula Calculate the pixel position of the deformation area. The process diagram is as follows Figure 3 The final defect sample result is shown in Figure 4 shown.

[0070] Step 2: Build a self-attention-based adversarial autoencoder texture surface defect detection framework.

[0071] The adversarial autoencoder network proposed in the present invention consists of three modules: autoencoder, adversarial network and classifier network. Figure 2As shown in the figure. The autoencoder network consists of three parts: encoder, decoder and intermediate layer. In the encoder, the convolution kernel size is 3, the stride and padding are both 1, and the activation function uses Leaky ReLU with a slope of 0.2. In the encoder, deconvolution is used to increase the size of the feature map. The intermediate layer network is built using four fully connected layers, the adversarial network uses the discriminator in the DCGAN network, and the classifier network is built using three fully connected layers.

[0072] The present invention proposes a self-attention framework combining channels and positions, such as Figure 2 In the channel self-attention module, first, the features output by the last layer of the encoder module are Input to the channel self-attention module, then re-adjust M A The shape is M A1 , where C represents the number of channels, N = H × W represents M A The number of features in M A1 With M A1 The channel attention map is obtained by multiplying the transposed matrix of Apply the softmax operation to get the final channel attention map X, and finally, apply the channel self-attention map to the feature map. In the position self-attention module, first, the channel self-attention module output M B The features of are input into three convolutional layers to generate three new feature maps M respectively. B1 , M B2 , and M B3 ,in Then, reshape it to Where C represents the number of channels, N = H × W represents M B Next, calculate M B2 and M B1 The cosine distance between the transposes of , and the softmax operation is applied to obtain the final position self-attention map S. Finally, the position self-attention map is applied to the feature map.

[0073] Step 3: Design the metric loss of input image and reconstructed image.

[0074] The present invention comprehensively considers the structural similarity between the reconstructed image and the input image and the attention map F of the input image, and designs the reconstruction loss as L re (i,j)=SSIM(I in (i,j),I re (i,j))*(1+C r *F(i,j)), where L re (i,j) represents the loss of the i,jth position of the image, I in represents the input image, Ire represents the reconstructed image, SSIM(·) represents the structural similarity loss, C r ∈{0,1} represents the classification result, and F(·) represents the attention map of the specified convolutional layer.

[0075] In addition, the present invention proposes an edge consistency loss in the gradient domain Make the edges of the input image consistent with the reconstructed image. Here, λ edge is a constant, U is the number of downsampling times, ↓u represents the downsampling operation, ||·|| F represents the F norm, σ() represents the weakly corrected linear unit, represents the gradient operation, and ⊙ represents the corresponding element multiplication operation.

[0076] Therefore, the final metric function is L = θ1L re +θ2L edge +θ3L cls +θ4L d , where θ i is a constant, L d To combat the loss, L cls is the cross entropy classification loss.

[0077] Step 4: Obtain pixel-level anomaly scores by calculating the multi-scale structural similarity between the test image and the reconstructed image, and then determine whether the test image has defects.

[0078] The present invention uses multi-scale structural similarity to calculate the anomaly map ASM (I in ,I re ); Here, I in represents the test image, I re Represents the reconstructed image. According to the abnormal image, by setting the segmentation threshold τ1 and the classification threshold τ2, the final pixel-level defect segmentation map and image-level defect classification category are obtained. Figure 5 The defect reconstruction image and detection results of the present invention are given. The first to fourth rows are the test image, the reconstructed image, the defect location result image and the true value image respectively.

Claims

1. A texture surface defect detection method based on self-attention adversarial autoencoder, characterized in that: The following steps are involved: Step 1: Based on the defect-free texture industrial product images, two defect generation methods based on color and deformation are built to generate defect data to simulate real defect samples; The construction of the color-based defect generation method includes the following processing steps: Step 1-1: Use Bezier curve to create a closed area; Step 1-2: Add random colors to the enclosed area; Step 1-3: Adjust the enclosed area to a random size; Step 1-4: Add the closed area to the random position of the defect-free image to complete the color-based method to generate defect data; the construction of the deformation-based defect generation method includes the following processing steps: Step 1-5: Randomly generate two points (p1, p2) and deformation radius r; Step 1-6: Calculate the deformation area based on p1, p2 and r; Step 1-7: According to the formula Calculate the pixel position of the deformation area and generate defect data based on the deformation method; Step 2: Build a self-attention-based adversarial autoencoder texture surface defect detection framework; The construction of the self-attention-based adversarial autoencoder texture surface defect detection framework includes the following processing steps: Step 2-1: Build the encoder network framework; Step 2-2: Build the decoder network framework; Step 2-3: Build a self-attention framework that combines channels and positions; Step 2-4: Use the discriminator in the DCGAN network to build the discriminator network framework; Step 2-5: Use four fully connected layer operations to build the middle layer network framework; Step 2-6: Use three fully connected layer operations to build the classifier network framework Step 3: Design the metric loss of input image and reconstructed image; Step 4: Obtain pixel-level anomaly scores by calculating the multi-scale structural similarity between the test image and the reconstructed image, and then determine whether the test image has defects.

2. According to claim 1, the method for detecting texture surface defects based on self-attention adversarial autoencoder is characterized in that: The construction of the encoder network framework described in step 2-1 includes the following processing steps: Step 2-1-1: Use the first five convolutional modules of the VGG-16 network as the encoding layer network; Step 2-1-2: After each convolution module, a ReLUs activation operation with a slope of 0.2 is performed, followed by a maximum pooling operation with a kernel size of 2*2 and a stride of 2.

3. According to claim 1, the method for detecting texture surface defects based on self-attention adversarial autoencoder is characterized in that: The construction of the decoder network framework described in step 2-2 includes the following processing steps: Step 2-2-1: For each convolution block in the encoder, there is a corresponding decoding block in the decoder; the first four decoding blocks contain a convolution layer and a deconvolution layer. The convolution layer uses a 3*3 convolution kernel, a step size of 1, and a padding of 1. The deconvolution layer uses a 4*4 convolution kernel, a step size of 2, and a padding of 1. After the convolution layer and the deconvolution layer, a ReLUs activation operation with a slope of 0.2 is used; Step 2-2-2: The fifth decoding block contains a convolution layer with a convolution kernel size of 3*3, a stride of 1, and a padding of 1.

4. According to claim 1, the method for detecting texture surface defects based on self-attention adversarial autoencoder is characterized in that: The construction of the channel self-attention framework described in step 2-3 includes the following processing steps: Step 2-3-1: Output the features of the last layer of the encoder module Input to the channel self-attention module; Step 2-3-2: Re-adjust M A The shape is M A1 , where C represents the number of channels, N = H × W represents M A The number of features in Step 2-3-3: Through M A1 With M A1 The channel attention map is obtained by multiplying the transposed matrix of Step 2-3-4: Then apply the softmax operation to obtain the final channel attention map X; Step 2-3-5: Use Formula M out =[X·M A1 ] R +M A Apply the channel self-attention map to the feature map, [·] R is an operation that reshapes the matrix.

5. According to the self-attention-based adversarial autoencoder texture surface defect detection method of claim 1, the construction of the self-attention framework includes the following processing steps: Step 2-3-6: Output M of the channel self-attention module B The features of are input into three convolutional layers to generate three new feature maps M respectively. B1 , M B2 , and M B3 ,in Reshape to Where C represents the number of channels, N = H × W represents M B The number of features in Step 2-3-7: Calculate M B2 and M B1 The cosine distance between the transposes of , and the position self-attention map S is obtained; Step 2-3-8: Use softmax operation to get the final position self-attention map; Step 2-3-9: Use Formula M out =[M B3 ·S] R +M B Apply the position self-attention map to the feature map.

6. The method for detecting texture surface defects based on self-attention adversarial autoencoder according to claim 1, characterized in that: The design of the metric loss of the input image and the reconstructed image described in step 3 includes the following processing steps: Step 3-1: Considering the structural similarity between the reconstructed image and the input image and the attention map F of the input image, the reconstruction loss is designed to be L re (i,j)=SSIM(I in (i,j),I re (i,j))*(1+C r *F(i,j)), where L re (i,j) represents the loss of the i,jth position of the image, I in represents the input image, I re represents the reconstructed image, SSIM(·) represents the structural similarity loss, C r ∈{0,1} represents the classification result, and F(·) represents the attention map of the specified convolutional layer; Step 3-2: Proposed edge consistency loss in gradient domain Make the edges of the input image consistent with the reconstructed image. Here, λ edge is a constant, U is the number of downsampling times, ↓u represents the downsampling operation, ||·|| F represents the F norm, σ() represents the weakly corrected linear unit, represents the gradient operation, ⊙ represents the corresponding element multiplication operation; Step 3-3: Design the final metric function as L = θ1L re +θ2L edge +θ3L cls +θ4L d , where θ i is a constant, L d To combat the loss, L cls is the cross entropy classification loss.

7. The method for detecting texture surface defects based on self-attention adversarial autoencoder according to claim 1, characterized in that: The determination of whether the test image described in step 4 is defective includes the following processing steps: Step 4-1: Use multi-scale structural similarity to calculate the anomaly map ASM (I in ,I re ); Here, I in represents the test image, I re represents the reconstructed image, Step 4-2: According to the abnormal image, by setting the segmentation threshold τ1 and the classification threshold τ2, the final pixel-level defect segmentation map and image-level defect classification category are obtained.

Images