Non-contrast defect detection method based on self-supervised reconstruction and multi-scale feature fusion

Abstract

The invention discloses a non-contrast defect detection method based on self-supervised reconstruction and multi-scale feature fusion, and relates to the technical field of image processing. The method comprises the following steps: acquiring an image data set only containing normal samples as a training set, and carrying out image preprocessing on image data in the training set; constructing a multi-scale memory enhanced auto-encoder network, wherein the multi-scale memory enhanced auto-encoder network comprises an encoder, a memory module and a multi-scale feature fusion decoder; training the memory module based on the training set, taking mean square error and structural similarity loss as reconstruction loss during training, and applying compactness constraint to the memory module; inputting an image to be detected into the trained multi-scale memory enhanced auto-encoder network to obtain a reconstructed image; and calculating a residual image between the to-be-detected image and the reconstructed image, and generating a defect segmentation image based on the residual image to realize pixel-level defect positioning of the to-be-detected image. According to the method, the detection capability on complex backgrounds and tiny defects is improved, and meanwhile, the calculation complexity is reduced.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a non-contrast defect detection method based on self-supervised reconstruction and multi-scale feature fusion. Background Technology

[0002] In industrial production, product quality control is a crucial step. Machine vision-based surface defect detection technology, due to its advantages such as non-contact operation, high efficiency, and high precision, is widely used in various industries including displays, semiconductors, textiles, and metal components.

[0003] Existing defect detection algorithms can be mainly divided into two categories: supervised and unsupervised. Supervised methods (such as classification and segmentation models based on convolutional neural networks) require a large number of labeled defect samples and normal samples for training. However, in real-world industrial scenarios, defect samples are often scarce and varied in form, making them difficult to collect, while the labeling cost is extremely high, which severely limits the applicability of supervised methods.

[0004] Unsupervised methods, also known as "uncompared" methods, aim to learn the distribution characteristics of normal samples through training only, and then detect defects by identifying anomalous regions that deviate from the normal pattern. Current mainstream methods include:

[0005] Reconstruction-based methods use autoencoders or generative adversarial networks (GANs) to learn reconstructions of normal samples. They assume that the model cannot reconstruct unseen (abnormal) patterns well. However, these methods suffer from "overgeneralization," meaning the model can sometimes reconstruct defects well, leading to missed detections.

[0006] Feature embedding-based methods model normal samples in the feature space of a pre-trained network (e.g., using Gaussian mixture models, K-nearest neighbors, etc.) and classify regions that deviate from this statistical model as defects. These methods are highly sensitive to the choice of feature space and have high computational complexity.

[0007] Therefore, existing non-contrast defect detection technologies generally suffer from poor adaptability to complex textures, inaccurate defect localization, susceptibility to noise interference, and high computational costs. There is an urgent need for a robust, accurate, and practical non-contrast defect detection solution. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide a detection method that can achieve high-precision pixel-level defect localization without defect samples and only based on normal samples for training. This method aims to improve the detection capability of complex backgrounds and small defects, while reducing computational complexity.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A non-contrast defect detection method based on self-supervised reconstruction and multi-scale feature fusion includes the following steps:

[0011] Step 1: Construct a dataset by obtaining an image dataset containing only normal samples as the training set, and perform image preprocessing on the image data in the training set.

[0012] Step 2: Construct a multi-scale memory-enhanced autoencoder network, including an encoder, a memory module, and a multi-scale feature fusion decoder;

[0013] The encoder includes a convolutional residual network and a downsampling layer, which are used to extract multi-level feature images of different scales from the input image.

[0014] The memory module is a memory bank of common features of the samples to be detected. It is used to store the potential feature prototypes of normal samples in the training set and obtain multiple sets of common feature images. The memory module includes several visual attention modules (ViT), which decompose the multi-level feature images output by the encoder into multiple sets of feature images through the attention modules, and match them with multiple sets of common feature images in the memory bank. The output is a fused feature image after weighted aggregation of the feature image of the input image and the common feature image, that is, the multi-scale reconstructed features of each level. This design forces the network to call the feature vectors that are closer to the standard from the memory bank for reconstruction, thus suppressing the reconstruction of abnormal patterns.

[0015] The multi-scale feature fusion decoder includes a convolutional residual network and a downsampling layer. The multi-scale feature fusion decoder fuses multi-scale reconstructed features from various levels of the memory module through skip connections, which helps to preserve more spatial detail information when reconstructing the image, thereby achieving more accurate defect localization.

[0016] Step 3: Train the multi-scale memory-enhanced autoencoder network based on the training set. During training, mean squared error and structural similarity loss are used as reconstruction losses to constrain the multi-scale memory-enhanced autoencoder network to learn the reconstruction of normal samples. At the same time, a compactness constraint is imposed on the memory module to encourage the memory bank to learn the common features of the most representative normal samples. The multi-scale memory-enhanced autoencoder network is trained end-to-end using only normal samples.

[0017] Step 4: Input the image to be detected into the trained multi-scale memory-enhanced autoencoder network to obtain the reconstructed image; calculate the residual map between the image to be detected and the reconstructed image, and generate a defect segmentation map based on the residual map to achieve pixel-level defect localization of the image to be detected.

[0018] Furthermore, in step 1, image preprocessing includes: normalization, size normalization, and data augmentation.

[0019] Further data augmentation processes include random cropping, rotation, and brightness jitter to increase the diversity of the training data.

[0020] Furthermore, in step 2, the encoder includes several coding units, each coding unit is composed of convolutional residual network blocks, and adjacent coding units are connected through max pooling layers.

[0021] Specifically, the convolutional residual network block is configured such that the input image is transformed to m dimensions through m 3×3 convolutions, where m is a configuration parameter;

[0022] The image is then processed by 3×3 convolution, batch normalization layer, and ReLU activation function to extract features. The extracted image features are then processed by 3×3 convolution and batch normalization layer to obtain the intermediate feature image of each coding unit. Based on residual connection, the input image is boosted to m dimensions by m 1×1 convolutions and then summed with the intermediate feature image. After passing through a ReLU activation function, the output is the m-dimensional coding feature image of each coding unit. At the same time, the coding feature image output by the coding unit is downsampled by max pooling layer to obtain the m-dimensional downsampled feature image as the input of the next coding unit. The last coding unit is directly connected to the memory module.

[0023] In this design, the downsampling factor of each max pooling layer is set to 2, the number of downsampling times is defined as n, and the encoder outputs a total of n+1 sets of m-dimensional encoded feature images at different scales, which are used as inputs to the memory module.

[0024] Furthermore, in step 2, the memory module includes a j-layer visual attention module, which is used to superimpose the k most recently input m-dimensional encoded feature images of the same scale in the dimensional direction and then perform pixel-wise average pooling to obtain the m-dimensional memory feature image of the current scale; where j and k are preset values; if there are less than k inputs, then the average pooling is directly performed on the encoded feature images of the current number of inputs.

[0025] The difference between the encoded feature image of the current input image and the memory feature image is calculated. The value of the position in the current encoded feature image corresponding to the position with the first a% of the difference or the position with the difference greater than the threshold b is replaced with the value in the memory feature image to obtain the multi-scale reconstructed features of each level.

[0026] Furthermore, in step 2, the multi-scale feature fusion decoder includes several decoding units, each of which consists of a convolutional residual network block, and adjacent decoding units are connected through an upsampling layer.

[0027] The input to the multi-scale feature fusion decoder is m-dimensional reconstructed features with n+1 scales increasing sequentially, where n is the downsampling number in the encoder. Starting from the smallest scale, the features are sequentially decoded by the decoding unit. The decoded features generated by the decoding process are then fed into the next decoding unit after being upsampled. The first decoding unit is directly connected to the memory module. Starting from the second decoding unit, its input is the fused feature of the output of the previous decoding unit and the same-scale reconstructed features fed in through a skip connection. The reconstructed image of the input image is obtained based on the output of the last decoding unit.

[0028] Furthermore, the visual attention module sequentially includes: a normalization layer, a multi-head attention layer, another normalization layer, and a multilayer perceptron; and a short connection is set between the input of the visual attention module and the output of the multi-head attention layer, and a short connection is set between the input of the second normalization layer and the output of the multilayer perceptron.

[0029] Furthermore, in step 3, the mean squared error is calculated as follows:

[0030] =

[0031] in, Indicates the input image. This represents the reconstructed image output. These are the image's height, width, and number of channels, respectively. , The input images are respectively How to output the image The position in the d-th channel dimension The pixel value at that location.

[0032] Furthermore, in step 3, the structural similarity loss is:

[0033]

[0034] in, For the input image and reconstructed image Structural similarity.

[0035] Furthermore, in step 3, the compactness constraint uses the Euclidean distance between feature vectors in the vector space to measure the similarity between two features, thereby controlling the number of features in the memory. The calculation method is as follows:

[0036]

[0037] Where x represents the newly calculated feature vector, and y represents the feature vector already existing in the memory.

[0038] Furthermore, in step 4, the residual plot is calculated using either absolute difference or squared difference.

[0039] Furthermore, in step 4, generating a defect segmentation map based on the residual map includes: performing Gaussian filtering-based post-processing on the residual map and using an adaptive thresholding algorithm to generate a binarized defect segmentation map.

[0040] The technical solution provided by this invention brings at least the following beneficial effects:

[0041] (1) No defective samples required: Training is performed using only normal samples, which solves the problem of scarce defective samples in industrial scenarios and has wider applicability.

[0042] (2) High detection accuracy: The innovative memory module can effectively store and recall normal mode, significantly suppress the reconstruction ability of defects, amplify the reconstruction residual, and make defects easier to identify.

[0043] (3) Precise positioning: The decoder structure, which combines multi-scale feature fusion, can reconstruct a normal background with high fidelity while retaining the sensitivity to the boundaries of small defects, thus achieving pixel-level precise segmentation.

[0044] (4) Strong robustness: It has good adaptability to image noise, lighting changes and complex texture backgrounds as well as different detection targets. In repeated detection with the same device and shooting scene, it fully suppresses background interference and reduces false detection rate.

[0045] (5) Balance between efficiency and performance: The backbone network and efficient memory design ensure high accuracy while meeting the real-time requirements of industrial testing. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is an overall flowchart of the non-contrast defect detection method provided in the embodiments of the present invention.

[0048] Figure 2 This is a schematic diagram of the network structure of the multi-scale memory-enhanced autoencoder in an embodiment of the present invention.

[0049] Figure 3 This is a schematic diagram of the structure of the residual convolution block and the ViT block in an embodiment of the present invention, wherein (3a) is the residual convolution block and (3b) is the ViT block.

[0050] Figure 4 This is a schematic diagram illustrating the working principle of the memory module in this embodiment of the invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described in detail and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Generally, the components of the embodiments of the present invention described and shown in the accompanying drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present invention.

[0052] This invention provides a non-contrast defect detection method based on self-supervised reconstruction and multi-scale feature fusion. The overall implementation process includes a training phase and a detection phase. Taking metal surface scratch detection as an example, the specific processes of the training and detection phases are described below:

[0053] 1. Training Phase:

[0054] (1-1) Construct a training dataset, collect a large number of normal metal surface images without any defects, and uniformly scale them to 512x512 pixels.

[0055] (1-2) Construct a multi-scale memory-reinforced autoencoder network, the structure of which is as follows: Figure 2 As shown:

[0056] Encoder: Employs a ResNet+ViT network pre-trained on ImageNet. See also Figure 3 In (3a), the backbone network input image is transformed to m dimensions through m 3×3 convolutions, and then features are extracted from the image through 3×3 convolution + BatchNorm + ReLU activation function. The feature image is then subjected to 3×3 convolution + BatchNorm to obtain an intermediate feature image. The residual connection increases the dimension of the input image to m dimensions through m 1×1 convolutions, sums it with the intermediate feature image, and then passes it through a ReLU activation function to obtain the final m-dimensional feature image of the current layer. Figure 2 In this architecture, the output feature map of each convolutional block is processed by the ViT attention layer and then input into the memory module. Simultaneously, the m-dimensional final feature image is downsampled using Maxpooling. This downsampled m-dimensional feature image serves as the input to the next residual connection network layer. This downsampling is performed n times, with each Maxpooling step having a downsampling factor of 2. A total of n+1 feature images are obtained for input to the memory module. In this embodiment, m is 3 and n is 3.

[0057] Memory Module: Constructs a j-layer ViT module, integrating the first k test feature images, and adjusting parameter a%. That is, the memory module consists of j layers of ViT modules. The feature images undergo feature transformation using an attention mechanism through the j-layer ViT module. The feature images of the previously input k test images are superimposed along the dimensional direction and then subjected to pixel-wise average pooling (if there are fewer than k inputs, average pooling is performed using the current number of input feature images), resulting in an m-dimensional memory feature image. The difference between the current test image's feature image and the memory feature image is calculated. The values ​​at the positions in the current test image's feature image corresponding to the first a% of the difference or the difference being greater than a threshold b (where a and b are hyperparameters and can be manually set) are replaced with the values ​​in the memory feature image, and the feature image is output to the decoder. In this embodiment, j is 3 and k is 10. The first a% filtering is used, with a set to 10.

[0058] like Figure 3 As described in (3b), each ViT module sequentially includes: a normalization layer (Layer Norm), a multi-head self-attention layer (Multi Head Self Attention), a normalization layer, and a multilayer perceptron (MLP); and a short connection is set between the input of the ViT module and the output of the multi-head self-attention layer, and a short connection is set between the input of the second normalization layer and the output of the multilayer perceptron.

[0059] Decoder: It adopts a symmetrical U-Net structure, gradually recovers the resolution through upsampling and convolution, and makes skip connections with the feature images output by the memory modules of the corresponding layers of the encoder.

[0060] like Figure 4 The memory module's processing includes: inputting a multidimensional feature map of the detection image; performing average pooling on the feature map after k signatures; comparing the current feature map with the pooled feature map; replacing the feature values ​​at the a% difference positions of the current feature map with the feature values ​​of the pooled feature map; and outputting the feature map optimized by memory matching.

[0061] (1-3) Training the network: The Adam optimizer is used, with MSE loss as the main reconstruction loss. Training is performed on a normal sample dataset until convergence. In this embodiment, the input image is set as... The output image is The input and output image sizes are both The mean squared error (MSE) is calculated as follows:

[0062] =

[0063] The structural similarity loss is:

[0064]

[0065] In the formula Indicates the input image and output image Structural similarity.

[0066] After feeding the training data into the model, the network model parameters are optimized by minimizing the following loss function:

[0067]

[0068] in Take values ​​of 1 and 0.1 respectively.

[0069] (1-4) Memory compactness constraint: Euclidean distance is used to measure the similarity between feature vectors. The Euclidean distance of feature vectors is calculated as follows:

[0070]

[0071] Where x represents the current input feature vector, and y represents the existing feature vectors in the memory. When the distance between two feature vectors is less than a threshold, they are considered to have extremely high similarity and can be classified as duplicate features, so they are not added to the memory. The specific calculation process is as follows:

[0072] For the current input feature vector x, each feature vector in the memory Y... The similarity between the currently input feature vector and the existing feature vectors in the memory is defined as follows:

[0073]

[0074] In this embodiment, the determination threshold is set to 0.1.

[0075] Testing phase:

[0076] (2-1) Input an image of the metal surface to be detected.

[0077] (2-2) Obtain the reconstructed image through the network. Ideally, normal areas are perfectly reconstructed, while defective areas with scratches are poorly reconstructed.

[0078] (2-3) Calculate the squared difference between the input image and the reconstructed image at each pixel to obtain the residual map.

[0079] (2-4) Gaussian filtering is applied to the residual map to smooth the noise, and then Otsu adaptive thresholding algorithm is used to generate a binary defect mask, where the white area is the location of the detected defect (scratch).

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0081] The above descriptions are merely some embodiments of the present invention. For those skilled in the art, various modifications and improvements can be made without departing from the inventive concept of the present invention, and all such modifications and improvements fall within the scope of protection of the present invention.

Claims

1. A non-contrast defect detection method based on self-supervised reconstruction and multi-scale feature fusion, characterized in that, Includes the following steps: Step 1: Construct a dataset by obtaining an image dataset containing only normal samples as the training set, and perform image preprocessing on the image data in the training set. Step 2: Construct a multi-scale memory-enhanced autoencoder network, including an encoder, a memory module, and a multi-scale feature fusion decoder; The encoder includes a convolutional residual network and a downsampling layer, which are used to extract multi-level feature images of different scales from the input image. The memory module is a memory bank of common features of the samples to be detected. It is used to store the potential feature prototypes of normal samples in the training set and obtain multiple sets of common feature images. The memory module includes several visual attention modules, which decompose the multi-level feature images output by the encoder into multiple sets of feature images through the attention modules, and match them with multiple sets of common feature images in the memory bank. The output is a fused feature image after weighted aggregation of the feature image of the input image and the common feature image, that is, the multi-scale reconstructed features of each level. The multi-scale feature fusion decoder includes a convolutional residual network and a downsampling layer. The multi-scale feature fusion decoder fuses multi-scale reconstructed features at each level of the memory module through skip connections. Step 3: Train the multi-scale memory-enhanced autoencoder network based on the training set. During training, mean squared error and structural similarity loss are used as reconstruction losses to constrain the multi-scale memory-enhanced autoencoder network to learn the reconstruction of normal samples. At the same time, compactness constraints are imposed on the memory modules to encourage the memory bank to learn the common features of normal samples. Step 4: Input the image to be detected into the trained multi-scale memory-enhanced autoencoder network to obtain the reconstructed image; calculate the residual map between the image to be detected and the reconstructed image, and generate a defect segmentation map based on the residual map to achieve pixel-level defect localization of the image to be detected.

2. The method as described in claim 1, characterized in that, In step 1, image preprocessing includes: normalization, size normalization, and data augmentation.

3. The method as described in claim 1, characterized in that, In step 2, the encoder includes several coding units, each coding unit is composed of convolutional residual network blocks, and adjacent coding units are connected through max pooling layers. Specifically, the convolutional residual network block is configured such that the input image is transformed to m dimensions through m 3×3 convolutions, where m is a configuration parameter; The image is then processed by 3×3 convolution, batch normalization layer, and ReLU activation function to extract features. The extracted image features are then processed by 3×3 convolution and batch normalization layer to obtain the intermediate feature image of each coding unit. Based on residual connection, the input image is boosted to m dimensions by m 1×1 convolutions and then summed with the intermediate feature image. After passing through a ReLU activation function, the output is the m-dimensional coding feature image of each coding unit. At the same time, the coding feature image output by the coding unit is downsampled by max pooling layer to obtain the m-dimensional downsampled feature image as the input of the next coding unit. The last coding unit is directly connected to the memory module. In this design, the downsampling factor of each max pooling layer is set to 2, the number of downsampling times is defined as n, and the encoder outputs a total of n+1 sets of m-dimensional encoded feature images at different scales, which are used as inputs to the memory module.

4. The method as described in claim 1, characterized in that, In step 2, the memory module includes a j-layer visual attention module, which is used to stack the most recently input m-dimensional encoded feature images of the same scale in the dimensional direction and then perform pixel-wise average pooling to obtain the m-dimensional memory feature image of the current scale; where j and k are preset values; if there are less than k inputs, then the average pooling is directly performed on the encoded feature images of the current number of inputs. The difference between the encoded feature image of the current input image and the memory feature image is calculated. The value of the position in the current encoded feature image corresponding to the position with the first a% of the difference or the position with the difference greater than the threshold b is replaced with the value in the memory feature image to obtain the multi-scale reconstructed features of each level.

5. The method as described in claim 1, characterized in that, In step 2, the multi-scale feature fusion decoder includes several decoding units, each of which consists of a convolutional residual network block, and adjacent decoding units are connected through an upsampling layer. The input to the multi-scale feature fusion decoder is an m-dimensional reconstructed feature with n+1 scales increasing sequentially, where n is the downsampling number in the encoder. Starting from the smallest scale, the features are sequentially decoded by the decoding unit. The decoded features generated by the decoding process are then fed into the next decoding unit after being upsampled. The first decoding unit is directly connected to the memory module. Starting from the second decoding unit, its input is the fused feature of the output of the previous decoding unit and the reconstructed features of the same scale fed in through a skip connection. The reconstructed image of the input image is obtained based on the output of the last decoding unit.

6. The method as described in claim 1, characterized in that, The visual attention module consists of, in sequence: a normalization layer, a multi-head attention layer, another normalization layer, and a multilayer perceptron; and a short connection is set between the input of the visual attention module and the output of the multi-head attention layer, and a short connection is set between the input of the second normalization layer and the output of the multilayer perceptron.

7. The method as described in claim 1, characterized in that, In step 3, the mean square error is calculated as follows: = ； in, Indicates the input image. This represents the reconstructed image output. These are the image's height, width, and number of channels, respectively. , The input images are respectively How to output the image The position in the d-th channel dimension The pixel value at that location.

8. The method as described in claim 1, characterized in that, In step 3, the structural similarity loss is: ； in, For the input image and reconstructed image Structural similarity.

9. The method as described in claim 1, characterized in that, In step 4, the residual plot is calculated using either absolute difference or squared difference.

10. The method as described in claim 1, characterized in that, In step 4, generating a defect segmentation map based on the residual map includes: performing Gaussian filtering-based post-processing on the residual map and using an adaptive thresholding algorithm to generate a binarized defect segmentation map.

Images