A surface defect detection method based on template comparison

Abstract

The application belongs to the technical field of image processing, and discloses a surface defect detection method based on template comparison. The method comprises the following steps: S1, collecting an industrial image of an object to be detected, constructing a training set, a verification set and a test set; constructing a surface defect detection network; S2, setting network parameters of the detection network; inputting templates and samples in the training set into the detection network, and outputting a segmentation graph; calculating a training loss to obtain the training loss, and adjusting the network parameters; S3, inputting data in the verification set into the detection network, calculating a training loss and an average loss, judging whether the average loss decreases, returning to step S2 when the average loss decreases, otherwise, the current network parameters are optimal network parameters; S4, inputting templates and samples in the test set into the detection network corresponding to the optimal grid parameters, obtaining a segmentation graph corresponding to each sample, so as to realize defect detection. Through the application, the feature expression of the sample is no longer directly learned, and only templates are collected, so that adaptation to a new batch is realized.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and more specifically, relates to a surface defect detection method based on template comparison. Background Technology

[0002] In recent years, in order to solve the problem of sample imbalance in the field of surface defect detection, methods such as anomaly detection and sample expansion have been widely studied and have made great progress.

[0003] However, flexible manufacturing systems are characterized by small batches and diverse product types. For new batches, there are insufficient positive samples for anomaly detection, and the sample distribution of new batches is difficult to predict, making sample expansion challenging. Therefore, there is an urgent need in this field to propose a model that can generalize to new batches.

[0004] Template-based single-sample learning methods have been widely studied. Siamese U-Net proposed a method based on healthy template comparison to achieve anomaly segmentation in CT images. DSSSNet proposed a template similarity measurement method to achieve anomaly segmentation in PCB images. However, these methods all assume that the difference between the template and the sample is only anomaly, but in reality, there is a lot of noise between the background features of the template and the sample. This noise is caused by displacement noise due to component positional offsets, deformation noise due to the inability to completely guarantee component consistency, and texture noise due to variations in component texture.

[0005] Although neural networks have strong feature fitting capabilities and texture noise is easy to eliminate, the biggest challenge in achieving defect segmentation based on template comparison single-sample learning is how to eliminate displacement and deformation noise due to the displacement and rotation invariance of convolutional neural networks. Summary of the Invention

[0006] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a surface defect detection method based on template comparison, which solves the problem that displacement elimination and deformation noise are difficult to eliminate in defect detection.

[0007] To achieve the above objectives, according to the present invention, a surface defect detection method based on template comparison is provided, the method comprising the following steps:

[0008] S1 acquires industrial images of the object to be inspected, selects images without defects as templates and images with defects as samples, and marks the defects in the samples at the pixel level to form labels, thereby obtaining the label corresponding to each sample. The template, samples and the labels corresponding to the samples form a dataset, which is divided into training set, validation set and test set; and a surface defect detection network for template comparison is constructed.

[0009] S2 sets the network parameters of the detection network; inputs the templates and samples in the training set into the detection network, and outputs the segmentation map corresponding to each sample; compares the segmentation map with the label corresponding to the sample to calculate the training loss, and adjusts the network parameters according to the training loss;

[0010] S3 inputs the data from the validation set into the detection network after adjusting the network parameters, calculates the training loss corresponding to the validation set, calculates the average loss using the training loss of the validation set, and determines whether the average loss decreases. If the average loss decreases, return to step S2; otherwise, the current network parameters are the optimal network parameters.

[0011] S4 inputs the templates and samples in the test set into the detection network corresponding to the optimal grid parameters to obtain the segmentation map corresponding to each sample in the test set, and obtains the defect information of the sample from the segmentation map, thereby realizing defect detection.

[0012] More preferably, in step S1, the detection network includes a feature extraction encoder and a feature fusion decoder, the feature extraction encoder includes a Siamese network and a dual attention mechanism, and the feature fusion decoder includes a recurrent residual attention mechanism.

[0013] More preferably, in step S2, the process of the detection network outputting the segmentation map is performed according to the following steps:

[0014] S21 Input the template and sample into the Siamese network which includes a multi-layer network structure, and each layer of the network structure outputs a feature map, thereby obtaining the feature map of each layer of the network corresponding to the template and sample respectively.

[0015] S22 simultaneously inputs the feature maps corresponding to the deepest layers of the template and sample into the dual attention mechanism to obtain the self-attention map and template-sample mutual attention map corresponding to the sample. Subtracting the self-attention map and mutual attention map from the mutual attention map yields a noise-free defect feature map.

[0016] S23 Subtract the feature maps of each network layer corresponding to the template and sample in step S21 to obtain the pairwise subtracted feature maps of each network layer.

[0017] S24 inputs the noise-free defect feature map and the pairwise subtraction feature map into the recurrent residual attention mechanism to obtain the removal mask corresponding to each layer of the network;

[0018] S25 performs feature fusion by multiplying the removal mask used for each network layer with the pairwise subtracted feature maps to obtain the required segmentation map.

[0019] More preferably, the twin network consists of two sets of network structures with identical structures and shared weights. This network structure is the same as ResNet, specifically ResNet-18, ResNet-34, ResNet-50, ResNet-101, or ResNet-152.

[0020] More preferably, in step S22, the self-attention map is performed according to the following relationship:

[0021]

[0022] Wherein, vectors Q, K, V are formed by convolving the feature maps corresponding to the deepest network layers of the samples with a 1×1 convolution W. q W k W v C1 is the number of channels in the vector.

[0023] More preferably, in step S22, the template-sample mutual attention map is performed according to the following relationship:

[0024]

[0025] in, The feature map corresponding to the deepest layer of the template network is convolved by W using a 1×1 convolution. k W v C1 is the number of channels in the vector.

[0026] More preferably, in step S24, the mask removal is performed according to the following relationship:

[0027]

[0028] Where, M0 = F d F d =F self -F cross It is a noise-free defect feature map, Up(·) is a linear upsampling operation, f(·) is a stretching function, Atten R (·) represents the residual attention mechanism, where W is a 1×1 convolution.

[0029] More preferably, the stretching function is performed according to the following relationship:

[0030] f(x) = log v ((v-1)·x+1)

[0031] Among them, v is used to adjust the degree of stretching; the larger the value, the greater the degree of stretching.

[0032] More preferably, the recurrent residual attention mechanism includes downsampling, upsampling, and a sigmoid activation layer connected in sequence. The downsampling includes max pooling and residual blocks. Max pooling is used to reduce the feature size by a factor of two, and the residual blocks are used for feature extraction. The upsampling includes linear upsampling, a merging operation, residual blocks, and a convolution operation. Linear upsampling is used to increase the feature size by a factor of two, the merging operation is used to fuse the extracted features with the original input features, the residual blocks are used to extract features, the convolution operation is used to reduce the number of channels, and the sigmoid activation layer is used to obtain a denoising mask with pixel values ​​from 0 to 1.

[0033] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0034] 1. This invention proposes a single-sample defect segmentation network based on template comparison. Instead of directly learning the feature representation of the sample, it learns how to compare the differences between the template and the sample. Therefore, for a new batch, defect segmentation can be achieved by collecting only the template. This network has a stronger generalization ability and can adapt to the characteristics of various small batches in flexible production lines.

[0035] 2. This invention proposes a dual attention mechanism, which, based on the spatial position-independent characteristic of the self-attention mechanism, removes position and deformation noise to obtain a noise-free defect feature map;

[0036] 3. This invention proposes a cyclic residual attention mechanism to obtain noise removal masks for different receptive fields in a deep-to-shallow manner, thereby achieving feature fusion. Attached Figure Description

[0037] Figure 1 This is a flowchart of a template comparison-based single-sample defect segmentation network constructed according to a preferred embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram of the overall framework of a template comparison-based single-sample defect segmentation network constructed according to a preferred embodiment of the present invention;

[0039] Figure 3 This is a schematic diagram of a twin network constructed according to a preferred embodiment of the present invention;

[0040] Figure 4 This is a schematic diagram of the dual attention mechanism framework constructed according to a preferred embodiment of the present invention;

[0041] Figure 5 This is a feature map of the dual attention mechanism constructed according to a preferred embodiment of the present invention;

[0042] Figure 6This is a schematic diagram of the recurrent residual attention mechanism framework constructed according to a preferred embodiment of the present invention;

[0043] Figure 7 This is a feature map of the cyclic residual attention mechanism constructed according to a preferred embodiment of the present invention;

[0044] Figure 8 These are test samples and detection results constructed according to preferred embodiments of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0046] like Figure 1 As shown, this invention proposes a template comparison-based single-sample defect segmentation network for detecting defective workpieces with apparent anomalies in industrial manufacturing processes. It mainly solves the problem of defect segmentation model generalization in flexible production lines with small batches and multiple types of samples for new batches with large sample variations.

[0047] Its main idea is: such as Figure 1 As shown, this invention no longer directly learns the feature representation of samples, but instead learns how to compare the differences between templates and samples. The main difficulty in comparison lies in the fact that, in addition to abnormal features, there is also background noise between templates and samples, and due to the translation, rotation, and other transformations of convolutional neural networks, displacement and deformation noise is very difficult to remove.

[0048] First, this invention designs a Siamese network to extract feature maps from the template and samples respectively. Then, inspired by the position-independent nature of self-attention mechanisms in natural language models, this invention proposes a dual-attention mechanism based on self-attention to extract noise-free defect feature maps. Finally, based on residual attention and recurrent neural networks, a recurrent residual attention mechanism is proposed to generate noise removal masks at different levels, achieving noise-free feature fusion and obtaining the segmentation result. Specifically, the following steps are included:

[0049] S1 collects industrial images from different batches, marks abnormal areas at the pixel level to form labels, and constructs a set of data consisting of templates (without defects) - samples (potentially defective) - labels, which are divided into training set, validation set, and test set according to a certain ratio. Defects include scratches, broken lines, dirt, damage, and missing or incorrect lines, etc., and labels are used to identify whether each pixel in the sample is a defect.

[0050] A surface defect detection network based on template comparison is constructed, including a feature extraction encoder and a feature fusion decoder. The feature extraction encoder contains a Siamese network and a dual attention mechanism. The input is a template and a sample, which is used to extract features from the template and the sample. The feature fusion decoder contains a recurrent residual attention mechanism. The input is features, which are fused to obtain the output map.

[0051] S2 sets the parameters of the detection network, and inputs the template-sample from the training set in step S1 into a Siamese network with a multi-layer network structure to obtain the feature maps of each layer of the template-sample respectively. The feature maps of the deepest template and sample are input into the dual attention mechanism to calculate the sample self-attention map and the sample-template mutual attention map, and subtract them to obtain a noise-free defect feature map. The noise-free defect feature map and the pairwise subtracted feature maps of the template and sample (e.g., the feature map of the template in the i-th layer of the network is subtracted from the feature map of the sample in the i-th layer of the network) are input into the recurrent residual attention mechanism to generate a noise removal mask for each layer in a deep-to-shallow manner. The noise removal mask and the pairwise subtracted feature maps are multiplied for feature fusion to obtain a simulated label, and finally the segmentation map is output.

[0052] The detection results are compared with the corresponding labels to calculate the training loss; the parameters of the test network are then adjusted.

[0053] S3 inputs the templates and samples from the validation set into the test network and calculates the training loss for iterative training of the network. If the average loss of the validation set no longer decreases within a certain number of training rounds, training stops, and the optimal parameterized model is obtained.

[0054] S4 inputs the test sample template-sample from the test set in S1 into the optimal parameterized model to obtain a segmentation map, which reflects information such as the location, size, and shape of the defect.

[0055] As a preferred embodiment of the present invention, the ratio of samples in the training set, validation set, and test set is 6:2:2.

[0056] like Figure 2 As shown, the surface defect detection network for template comparison includes a feature extraction encoder and a feature fusion decoder. The feature extraction encoder contains a Siamese network and a dual attention mechanism, while the feature fusion decoder contains a recurrent residual attention mechanism.

[0057] like Figure 3 As shown, the Siamese network consists of two sets of network structures with identical structures and shared weights. The network structures are the same as those of ResNet, such as ResNet-18, ResNet-34, ResNet-50, ResNet-101, and ResNet-152. The network structure can be adjusted to take into account both image complexity and computational real-time performance.

[0058] like Figure 4 As shown, the dual attention mechanism includes five 1×1 convolutional kernels. Further, the specific steps are as follows:

[0059] Step 1: Input the template and sample into the Siamese network to obtain the deepest feature map of the template. The deepest feature map of the sample is Through 1×1 convolution W q W k W v Obtain vector

[0060] Q = W q F x

[0061] K = W k F x

[0062]

[0063] V = W v F x

[0064]

[0065] Step 2: Adjust the vector size Where M = H × W;

[0066] Step 3: Obtain the sample self-attention feature map

[0067]

[0068] Step 4: Obtain the template-sample mutual attention feature map

[0069]

[0070] Step 5: Noise-free defect feature map Through self-attention feature map F self Mutual attention feature map F cross Subtraction:

[0071] F d =F self -F cross

[0072] like Figure 5As shown, compared with the feature map extracted by the Siamese network, the dual attention mechanism effectively eliminates displacement and deformation noise. Both the Siamese network and the dual attention mechanism can effectively eliminate texture noise, demonstrating the powerful feature representation capability inherent in neural networks.

[0073] like Figure 6 As shown, the recurrent residual attention mechanism contains a 1×1 convolution W, and the residual attention mechanism Atten R The upsampling Up and the stretching function f, further, the specific steps are as follows:

[0074] Step 1: Input the template and samples into the Siamese network, where the deepest layer features are input into a dual attention mechanism to obtain a noise-free defect feature map F. d The four shallower feature maps are subtracted from each other to obtain feature map F1, which is obtained from deep to shallow. d ,

[0075] Step 2: Extract the deepest features F d Upsampling, followed by a stretching function, yields f(Up(F)). d Subtraction of feature layer F1 d After 1×1 convolution, the residual attention mechanism obtains Attenuation. R (WF1 d The noise removal mask M1 is:

[0076] M1=f(Up(F d ))×Atten R (WF1 d )

[0077] Step 3: Connect M1 and As input, the second step of the process is repeated to obtain the noise removal mask M2:

[0078] M2 = f(Up(M1)) × Atten R (WF1 d )

[0079] Repeat the above process to obtain M3 and M4 respectively.

[0080] Furthermore, the residual attention mechanism comprises three parts arranged sequentially: downsampling, upsampling, and a sigmoid activation layer. The downsampling part includes a max-pooling layer with a stride of 2 and a residual block. The upsampling part includes a linear upsampling section with a magnification of 2, a merging operation, a residual block, and a 1×1 convolution operation. The merging operation is used to fuse the original input features and the extracted features, avoiding information loss. The residual block includes concatenated 3×3 convolution operations, batch normalization, and the ReLU activation function.

[0081] Furthermore, the stretching function is:

[0082] f(x) = log v ((v-1)·x+1)

[0083] The value of 'v' is used to adjust the degree of stretching; the larger the value, the greater the stretching.

[0084] like Figure 7 As shown, the noise increases from deep to shallow in the feature map, and the noise removal mask generated using only the residual attention mechanism is not ideal. Using a noise-free defect feature map as a guide to generate a noise removal mask from shallow to deep results in a significant improvement. Further incorporating a stretching function further enhances the noise removal mask's effectiveness, making it nearly ideal.

[0085] like Figure 8 As shown, the network proposed in this invention achieves excellent detection results.

[0086] The present invention will be further described below with reference to specific embodiments.

[0087] This embodiment uses the detection of optical communication devices as an example to illustrate the method of the present invention.

[0088] This embodiment of a template comparison-based single-sample defect segmentation network includes the following steps:

[0089] (1) Create the dataset;

[0090] This embodiment collected 918 images of optical communication devices, forming a dataset ODCs. The dataset includes five types of defects: base damage, base scratches, component breakage, component contamination, and errors / missing lines. The image format is BMP. Each image was normalized to a uniform pixel size of 256×256 (unit: pixels). The images were divided into training, validation, and test sets, as shown in Table 1.

[0091] Table 1 Dataset ODCs

[0092] type training set Validation set test set total Base damage 36 12 12 60 Base scratches 17 5 5 27 Component damage 144 48 48 240 Component contamination 225 75 75 375 More mistakes, fewer lines 130 43 43 216

[0093] (2) Single-sample defect segmentation network;

[0094] In this embodiment, a single-sample defect segmentation network is constructed on the PyTorch 1.4 deep learning framework.

[0095] (3) Dataset training and model optimization;

[0096] (3-1) Write a Python program to randomly sort the training samples and distribute them evenly. The batch size for batch training is 4 (unit: sheets).

[0097] (3-2) Training images are input into the model in batches. After the sum of the gradient descent values ​​of all samples in a batch is calculated, the weights are updated once, until all batches are updated. The trained model is evaluated using a validation set to obtain an evaluation value (in this example, a loss function is used). If the evaluation value is greater than the current maximum evaluation value, the current parameterized model is stored and the next iteration is performed; if it is less than the maximum evaluation value, the next iteration is performed directly. The number of consecutive iterations less than the maximum evaluation value is accumulated. If the number of iterations is greater than 50, the training ends. The number of training sample iterations is set to 500. Training ends when the number of iterations is completed. The optimized parameterized model is named GWNet-ODCs-model.

[0098] (3-4) The trained model was tested using the input test set, and various metrics were obtained. The results are listed in Table 3. The model was tested on GWNet-ODCs-model, and the test set accuracy was 0.9070, the recall was 0.8891, the F1 score was 0.8980, and the mIoU was 0.8074, achieving very accurate anomaly identification.

[0099] Table 3 Overall Evaluation Indicators of BNN-SDD Model

[0100] network accuracy Recall rate F1 score mIoU GWNet 0.7771 1.9329 0.8980 0.8074

[0101] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for surface defect detection based on template contrast, characterized in that, The method comprises the following steps: S1: Collecting an industrial image of an object to be detected, selecting an image without defects as a template, an image with defects as a sample, marking the defects in the sample at the pixel level to form a label, thereby obtaining a label corresponding to each sample, the template, the sample, and the label corresponding to the sample forming a data set, dividing the data set into a training set, a validation set, and a test set; constructing a template comparison surface defect detection network; S2: Setting network parameters of the detection network; inputting the template and the sample in the training set into the detection network, outputting a segmentation map corresponding to each sample; comparing the segmentation map with the label corresponding to the sample to obtain a training loss, and adjusting the network parameters according to the training loss; S3: Inputting data in the validation set into the detection network with the adjusted network parameters, calculating a training loss corresponding to the validation set, calculating an average loss using the training loss of the validation set, and determining whether the average loss decreases; if the average loss decreases, returning to step S2; otherwise, the current network parameters are the optimal network parameters; S4: Inputting the template and the sample in the test set into the detection network corresponding to the optimal network parameters, obtaining a segmentation map corresponding to each sample in the test set, and obtaining defect information of the sample from the segmentation map, thereby realizing defect detection; In step S1, the detection network comprises a feature extraction encoder and a feature fusion decoder, the feature extraction encoder comprises a twin network and a double attention mechanism, and the feature fusion decoder comprises a cyclic residual attention mechanism; In step S2, the process of outputting the segmentation map by the detection network is performed according to the following steps: S21: Inputting the template and the sample into the twin network comprising a plurality of network structures, each network structure outputting a feature map, thereby obtaining a feature map corresponding to each network of the template and the sample respectively; S22: Inputting the feature maps corresponding to the deepest network of the template and the sample into the double attention mechanism at the same time, obtaining a self-attention map corresponding to the sample and a template-sample mutual attention map, and subtracting the self-attention map and the mutual attention map to obtain a defect feature map without noise; S23: Subtracting the feature maps corresponding to each network of the template and the sample in step S21 from each other, thereby obtaining a subtracted feature map corresponding to each network; S24: Inputting the defect feature map without noise and the subtracted feature map into the cyclic residual attention mechanism, thereby obtaining a removal mask corresponding to each network; S25: Multiplying the removal mask corresponding to each network with the subtracted feature map to perform feature fusion, thereby obtaining the required segmentation map.

2. A template contrast based surface defect detection method as claimed in claim 1, wherein, The twin network is a network structure with the same structure and shared weights, and the network structure is the same as ResNet, which is ResNet-18, ResNet-34, ResNet-50, ResNet-101, or ResNet-152.

3. A template contrast based surface defect detection method as claimed in claim 1, wherein, In step S22, the self-attention map is calculated according to the following relationship: Wherein, the vector The feature map corresponding to the sample deepest network passes through Convolution Obtained, The channel number of the vector.

4. The method of claim 1, wherein the method is a template-matching based surface defect detection method. In step S22, the template-sample mutual attention map is calculated according to the following relationship: wherein, The feature map corresponding to the deepest network of the template is obtained by convolution obtained, the number of channels of the vector.

5. A template contrast based surface defect detection method as claimed in claim 1, wherein, In step S24, the removal mask is performed according to the following relationship: wherein, , is a noise-free defect feature map, is a linear up-sampling operation, is a stretching function, is a residual attention mechanism, is a convolution.

6. A template contrast based surface defect detection method as claimed in claim 5, wherein, The stretching function is performed according to the following relationship: wherein The value is used to adjust the degree of stretching. The larger the value, the greater the degree of stretching.

7. A template contrast based surface defect detection method as claimed in claim 1, wherein, The cyclic residual attention mechanism comprises sequentially connected down-sampling, up-sampling and sigmoid activation layers, wherein the down-sampling comprises maximum pooling and a residual block, the maximum pooling is used for halving the feature size, and the residual block is used for feature extraction; the up-sampling comprises linear up-sampling, a merging operation, a residual block and a convolution operation, the linear up-sampling is used for doubling the feature size, the merging operation is used for fusing the extracted features and the original input features, the residual block is used for extracting features, and the convolution operation is used for reducing the number of channels; and the sigmoid activation layer is used for obtaining a de-noising mask with pixel values of 0-1.

Images