A method and system for detecting surface defects of an object

Abstract

This invention discloses a method and system for detecting surface defects of objects, belonging to the field of industrial visual anomaly detection in machine vision. It improves the real-time image segmentation network Fast-SCNN and constructs a surface defect detection network, which includes an encoder and a decoder. The encoder includes a convolutional module, a detail branch, and a semantic branch. The decoder includes a pyramid pooling module (PPM) and a global context upsampling module (GCU). Using this surface defect detection network, surface defects of objects can be detected in real time, quickly, and accurately.

Description

Technical Field

[0001] This invention belongs to the field of industrial vision anomaly detection in machine vision, specifically relating to a real-time and accurate method and system for detecting surface defects of objects. Background Technology

[0002] Automated visual defect detection in industrial production processes has a wide and urgent application demand, which can improve enterprise production efficiency, save costs, and bring significant socio-economic benefits. Surface defect detection is an important category of industrial defect detection, aiming to detect defective (abnormal) areas on the surface of workpieces. Typical visual defect detection is deployed on production lines, thus requiring high accuracy and real-time performance. Currently, most factories still rely on manual inspection, which is time-consuming and costly. Traditional visual defect detection methods are generally based on texture analysis, such as gradient operators and Canny edge detection operators. However, traditional methods lack universality for different tasks, requiring redesigned feature extraction methods for specific application scenarios.

[0003] In recent years, with the outstanding performance of deep learning in various computer vision tasks, some works have applied methods based on convolutional neural networks (CNNs) to different industrial scenarios, such as surface defect detection of steel strip magnetic tiles, train tracks, wire insulators, road cracks, and PCB boards. These methods have good robustness to surface regions of different objects and propose corresponding functional modules for specific problems such as small target defects, defects, and low background contrast. Most of these methods are based on object detection frameworks and cannot achieve pixel-level detection, which is a disadvantage for high-precision applications. Currently, there is a lack of pixel-level labeled surface defect detection datasets, with only the NEU-Seg dataset for hot-rolled steel surface defects proposed by Northeastern University and the Magnetic-tile-defect-datasets dataset for magnetic tiles proposed by the Chinese Academy of Sciences. These two datasets are relatively small, containing only about 1,000 images. Summary of the Invention

[0004] The purpose of this invention is to propose an accurate and real-time pixel-level detection method and system for surface defects. This invention improves upon the real-time image segmentation network Fast-SCNN and proposes a global context module and boundary and semantic auxiliary supervision tasks based on the characteristics of surface defect detection, to achieve accurate and rapid detection.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] This invention provides a method for detecting defects on the surface of an object, comprising the following steps:

[0007] A network for detecting surface defects of objects is constructed, which includes an encoder and a decoder; the encoder includes a convolutional module, a detail branch, and a semantic branch; the decoder includes a pyramid pooling module (PPM) and a global context upsampling module (GCU).

[0008] The training steps for the object surface defect detection network include:

[0009] The training set containing images of different defect types is input into the object surface defect detection network. The encoder's convolution module extracts the low-level feature maps of the training set images and inputs them into the decoder's PPM module.

[0010] The encoder's detail branch and semantic branch perform two auxiliary tasks. The detail branch performs a boundary detection task, extracting defect boundary features from the training set images. The semantic branch performs a multi-label classification task, predicting the types of defects contained in the images, generating ground truth labels, and supervising the boundary features. The boundary features are then used as high-level feature maps and input into the decoder.

[0011] The decoder's GCU module fuses the high-level feature map with the low-level feature map, and then uses bilinear interpolation upsampling and argmax to obtain the defect type detection result from the fused feature map;

[0012] The auxiliary task uses a binary cross-entropy loss function to optimize network parameters and complete training.

[0013] The trained object surface defect detection network is used to detect object surface defects, acquire and process object surface defect images, and identify the types of object surface defects.

[0014] Another aspect of the present invention proposes an object surface defect detection system, including an object surface defect detection network. The encoder includes a convolutional module, a detail branch, and a semantic branch; the decoder includes a pyramid pooling module (PPM) and a global context upsampling module (GCU); after the object surface defect detection network is trained, it is used to detect object surface defects, acquire object surface defect images and process them, and identify the types of object surface defects.

[0015] The training steps for this object surface defect detection network include:

[0016] The training set containing images of different defect types is input into the object surface defect detection network. The encoder's convolution module extracts the low-level feature maps of the training set images and inputs them into the decoder's PPM module.

[0017] The encoder's detail branch and semantic branch perform two auxiliary tasks. The detail branch performs a boundary detection task, extracting defect boundary features from the training set images. The semantic branch performs a multi-label classification task, predicting the types of defects contained in the images, generating ground truth labels, and supervising the boundary features. The boundary features are then used as high-level feature maps and input into the decoder.

[0018] The decoder's GCU module fuses the high-level feature map with the low-level feature map, and then uses bilinear interpolation upsampling and argmax to obtain the defect type detection result from the fused feature map;

[0019] The auxiliary task employs a binary cross-entropy loss function, which is used to optimize network parameters and complete training.

[0020] For the above-described methods and systems, the present invention also proposes the following preferred technical means:

[0021] Preferably, the defect types include three categories: oil stains, scratches, and spots.

[0022] Preferably, the encoder's convolution module consists of six MobileNetV3 modules.

[0023] Preferably, the encoder extracts high-level feature maps at 1 / 8 and 1 / 16 resolution of the image, and inputs the high-level feature maps into the decoder through skip connections of matrix element multiplication and addition; during the training phase, the high-level feature maps at 1 / 8 resolution contain defect boundary features for detail branch detection.

[0024] Preferably, the GCU module includes two global context modules (GC), two upsampling modules (Upsample), and two group convolutional modules (GroupConv), in the order of GC+Upsample+GroupConv+Upsample+GroupConv+GC. The first Upsample fuses high-level feature maps with a resolution of 1 / 16 by element-wise addition, and the second Upsample fuses high-level feature maps with a resolution of 1 / 8 by element-wise multiplication.

[0025] Preferably, the detail branch is composed of convolution, batch normalization, ReLU function, convolution, and bilinear interpolation upsampling network.

[0026] Preferably, the semantic branch is composed of convolution, batch normalization, ReLU function, convolution, global average pooling, and linear layer network.

[0027] Preferably, the method for calculating the ground truth label is as follows: the second gradient of the image is calculated using the Laplacian operator, and three boundary detail feature maps of different scales are calculated using three Laplacian convolutions of different lengths; then the three detail feature maps are upsampled to the original size of the input image, and then the three boundary detail features are concatenated and the weights are dynamically adjusted through two-dimensional convolution to obtain the ground truth label of the boundary.

[0028] Preferably, the Laplace operator is used for discrete computation using an 8-neighborhood operator template.

[0029] Technical effects of the present invention:

[0030] This invention proposes a global context module and auxiliary tasks for defect detection tasks, which effectively improve the performance and speed of detection. This invention achieves efficient detection of surface defects, striking a good balance between speed and accuracy. The method of this invention achieves optimal performance and the fastest detection speed on the mobile phone screen surface defect dataset MSD, is the best real-time method on the magnetic tile defect dataset Magnetic-tile-defect-datasets, and also achieves competitive performance on the hot-rolled steel dataset. Attached Figure Description

[0031] Figure 1 This is an overall framework diagram of the object surface defect detection network in this embodiment of the invention;

[0032] Figure 2 This is a partial example of the dataset created in this embodiment of the invention;

[0033] Figure 3 This is the global context upsampling module proposed in the embodiments of the present invention;

[0034] Figure 4 This is the header structure of the auxiliary task branch in this embodiment of the invention;

[0035] Figure 5 This is the truth value generation process for the boundary detection auxiliary task in this embodiment of the invention;

[0036] Figure 6 This is the truth value generation process for the multi-label classification auxiliary task in this embodiment of the invention;

[0037] Figure 7 These are some examples of experimental results from embodiments of the present invention. Detailed Implementation

[0038] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings.

[0039] 1. A dataset of mobile phone surface defects was created and proposed.

[0040] To address the lack of defect detection datasets, this embodiment proposes a pixel-level annotated mobile phone screen surface defect dataset, Mobile phone screen Surface Defect (MSD). Industrial cameras were used for data acquisition. The defects were artificially created to simulate common defects encountered in real industrial environments, such as... Figure 2 As shown, the defect types include three categories: oil stains, scratches, and spots, with 400 images for each category, for a total of 1200 images in the dataset. The image resolution is 1920*1080; the dataset is divided into training, validation, and test sets, with an image ratio of 6:2:2. The dataset uses the common PASCAL VOC format.

[0041] 2. An improved surface defect detection network framework based on Fast-SCNN

[0042] This embodiment employs the object surface defect detection method and system proposed in this invention. The overall network framework of the object surface defect detection network on which it is based is as follows: Figure 1 As shown. Given an image containing defects as input, the network's encoder and decoder produce a defect-segmented image of the same size as the original image. The encoder consists of a series of convolutional operations (see...). Figure 2 The algorithm consists of 1 Conv2D block, 2 DSConv blocks, and 6 MobileNetV3 blocks. It downsamples the image to obtain high-level (semantic) feature maps of defects. The decoder restores the image to its original size, including the commonly used pyramid pooling module (PPM) and the global context upsampling module (GCU) proposed in this embodiment. Finally, the feature maps are upsampled by a factor of 8 using bilinear interpolation and argmax is used to obtain the final result.

[0043] 3. Global Context Upsampling (GCU) Module

[0044] The structure of the GCU module is as follows: Figure 2 and Figure 3 As shown, this embodiment utilizes high-level feature maps at 1 / 8 and 1 / 16 resolutions from the encoder, employing skip connections to preserve more spatial detail. Skip connections are used to input the high-level feature maps from the encoder into the decoder to maintain the spatial detail of the image, resulting in more accurate segmentation. The high-level feature maps are progressively upsampled (i.e., ⊕ and ⊙ in the figure) and fused with the detailed information from the low-level feature maps. For example... Figure 7As shown, when inputting images with three types of defects—oil stains, scratches, and spots—the baseline method Fast-SCNN (refer to "Fast-SCNN: Fast semanticsegmentation network", Poudel RPK, Liwicki S, Cipolla R. in Proc. BMVC 2019) produces discontinuous predictions and lacks the ability to establish long-range context. Therefore, this embodiment employs a Global Context (GC) module (refer to "Gcnet: Non-local networks meet squeeze-excitation networks and beyond", Cao Y, Xu J, Lin S, et al. in Proc. ICCVW 2019) and inserts it at the beginning and end of the GCU module to capture long-range relations from multi-scale features, obtaining semantic relevance between pixels of a large range of objects in the image. The GC module is a simplified, lightweight self-attention (non-local) module. Due to the high similarity of attention maps at different query locations, the GC module computes a global attention map for all query locations. It can resolve the issues of inter-class similarity and intra-class differences, thereby making the predicted defect regions more complete. The GCU module comprises the following modules in the following order: GC module + Upsample module + GroupCovn module + Upsample + GroupCovn + GC module. The effects achieved by this invention are as follows: Figure 7 As shown, the defect detection performance of the present invention is superior to that of the baseline method.

[0045] The computation process of the Global Context (GC) module used in this embodiment. Given a feature map of a CNN. N p =N·W represents the number of positions in the feature map. x and z represent the input and output of the GC block, respectively, and W k and W v This represents the linear transformation matrix. The GC module calculates it as follows:

[0046]

[0047] The formal representation of the global context upsampling module calculation process proposed in this embodiment is as follows:

[0048]

[0049] Where Conv represents grouped convolution followed by BN and ReLU, Up represents bilinear interpolation, and GC represents the GC module. It represents the feature maps from the encoder stage to different resolutions. Supervised by a boundary detection task, it incorporates more spatial details. This embodiment uses element-wise multiplication to fuse more details.

[0050] 4. Boundary and semantic-assisted supervision

[0051] This embodiment designs two auxiliary tasks: boundary detection for detail branches and multi-label classification for semantic branches. Figure 4 The network is completed to supervise its performance on specific tasks. An auxiliary task is used during the training phase to improve the accuracy of detection results without affecting the inference speed during runtime. Surface defects mostly have obvious boundaries and are sensitive to defect boundary (edge) features. This embodiment uses a boundary detection task for detail prediction. This binary segmentation task captures effective features of defect boundaries, which can sharpen and refine predictions. In this embodiment, the boundary detection head is inserted into a 1 / 8 resolution feature map to generate a feature map containing spatial details. This feature map is supervised by the generated ground truth labels. Then, the learned detailed features are fused into the context-rich feature map of the decoder using element-wise matrix multiplication.

[0052] Different surface defects share certain similarities in their local features. Therefore, this embodiment employs a multi-label classification task to progressively guide defect classification, providing more semantic information for subsequent layers. The module corresponding to this task is inserted before the Pyramid Pooling (PPM) module to predict the types of defects contained in the image. When there is only one type of defect in the image, this task can be considered image classification and can be performed well. Even when multiple defects exist in the image, this task can still provide supporting information. The detailed branch header and semantic branch header of the auxiliary task are as follows... Figure 4 As shown. The detail branch header consists of consecutive 3×3 convolutions, batch normalization, ReLU function, 1×1 convolution, and 8x bilinear interpolation upsampling; the semantic branch header consists of consecutive 3×3 convolutions, batch normalization, ReLU function, 3×3 convolution, global average pooling, and a linear layer. Figure 5 As shown, this embodiment generates ground-truth labels (also known as supervision labels) for the auxiliary task from the semantic segmentation labels. For the ground-truth labels of the binary boundary detection task, the Laplacian operator is used to find the defect boundaries. Figure 6As shown, the unique values ​​of the semantic segmentation label image are counted to obtain class vector labels for multi-label classification. This involves removing all duplicate numbers from an image (e.g., using the `np.unique` function), marking the positions of the remaining numbers as 1, and marking the positions of numbers not present in the image as 0, thus obtaining a classification label. During the training phase, the auxiliary task employs binary cross-entropy loss. The method proposed in this embodiment utilizes the auxiliary task as a multi-task learning problem.

[0053] Labels representing semantic segmentation, and labels G for multi-label classification auxiliary tasks. s The generation of can be formally represented as:

[0054] G S =Unique(G out (3)

[0055] The Unique feature identifies unique values ​​in an image and is implemented using numpy.unique.

[0056] In this embodiment, the generation of ground truth labels for the supervised boundary detection task uses the Laplacian operator. The Laplacian operator calculates the second-order gradient of the image, and its formula is as follows:

[0057]

[0058] For discrete calculations of the image, this embodiment uses an 8-neighborhood operator template:

[0059]

[0060] This embodiment uses three Laplacian convolutions of different lengths (1, 2, 4) to obtain multi-scale boundary detail information. This embodiment upsamples three detail feature maps to the original size of the input image, then concatenates the three features and performs a 1×1 2D convolution to dynamically adjust the weights, resulting in a better boundary label. The above process can be formally represented as:

[0061] G B =Conv([Laplace1(G out ),Laplace2(G out ),Laplace4(G out )])

[0062] Where [,] indicates channel-by-channel concatenation, and Conv represents 1×1 convolution. Using Laplacian convolution with three strides can obtain object boundaries at different scales, which is more effective for images containing objects with multiple scale variations. The three feature maps are concatenated and then subjected to a learnable 2D convolution to obtain the final ground truth labels for the boundaries. In manual annotation, object boundary annotations are often inaccurate, leading to errors in the object boundaries obtained by Laplacian convolution. The 1×1 2D convolution operation can adaptively learn the ground truth labels and dynamically adjust the boundary labels to achieve better detection results.

[0063] During the training phase, the auxiliary task employs binary cross-entropy loss. The network outputs three predictions, and the loss function consists of three parts:

[0064]

[0065] Where L out It is the loss due to segmentation. P B ,P S This represents the output of the auxiliary task. W1 and W2 represent the weights of the losses for boundary detection and multi-label classification, respectively. N is the number of pixels in the image. i ∈{0,1} is the label of the pixel, p i ∈[0,1] is the probability that the network predicts the current pixel as a defect. The base of the logarithm is e.

[0066] 5. Model Acceleration

[0067] For surface defect detection tasks, defect images are generally relatively simple, requiring fewer convolutional blocks to extract high-level features. Fast-SCNN uses nine bottleneck residual blocks for high-level feature extraction. This embodiment replaces them with six modules introduced in MobileNetV3. This module uses channel attention, which can improve the contrast between defects and the background. TensorRT is also used to accelerate the model in this embodiment to meet the real-time requirements of industrial applications.

[0068] Experiments have demonstrated the advantages of the method of this invention.

[0069] To verify the effectiveness of the method proposed in this invention, experiments were conducted on the publicly available surface anomaly detection datasets NEU-Seg, MTD, and the self-built mobile phone screen anomaly dataset MSD.

[0070] For the NEU-Seg data set, see H.Dong, K.Song, Y.He, J.Xu, Y.Yan, and Q.Meng, "PGANet: Pyramid feature fusion and global context attention network for automated surface defect detection," IEEE Transactions on Industrial Informatics, vol.16, no.12, pp.7448–7458, 2019.

[0071] For the MTD dataset, see Y. Huang, C. Qiu, and K. Yuan, “Surface defect saliency of magnetic tile,” The Visual Computer, vol. 36, no. 1, pp. 85–96, 2020.

[0072] (1) Experimental Setup The experiment was implemented using the PyTorch 1.9 deep learning framework. All input images were scaled (0.5-2.0) and cropped to 450×450 pixels. During training, the batch size was 16, and the training run lasted for 150 epochs. The Adam optimization algorithm was used to optimize the network, with the initial learning rate and weight decay coefficient set to 10. -4 and 10 -5 The dataset was divided into a training:validation:test ratio of 6:2:2.

[0073] (2) Experimental Results

[0074] The table below shows the comparison of our experimental method with other state-of-the-art methods (conventional semantic segmentation networks and real-time semantic segmentation networks) on three datasets. This experiment lists the number of parameters, mean intersection-union ratio (mIoU), and running speed (FPS) for each method.

[0075] Table 1 Comparison of our proposed method with other state-of-the-art methods on three datasets.

[0076]

[0077]

[0078] On the MSD dataset, the proposed method outperforms all other methods in both performance and speed, achieving 90.2% mIoU and 135.0 FPS. On the MTD dataset, the proposed method achieves the fastest running speed and outperforms all real-time semantic segmentation methods, while also remaining competitive in conventional semantic segmentation networks. On the NEU-Seg dataset, it achieves the fastest running speed and outperforms the baseline method by 1.1% mIoU. These experimental results demonstrate that the proposed method achieves superior performance on all three datasets, validating the effectiveness of the proposed method.

[0079] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions made by those skilled in the art to the technical solutions of the present invention should be covered within the protection scope of the present invention, which is defined by the claims.

Claims

1. A method for detecting defects on the surface of an object, characterized in that, Includes the following steps: Construct a network for detecting surface defects of objects, which includes an encoder and a decoder; the encoder includes a convolutional module, a detail branch, and a semantic branch. The decoder includes a pyramid pooling module (PPM) and a global context upsampling module (GCU). The training steps for the object surface defect detection network include: The training set containing images of different defect types is input into the object surface defect detection network. The encoder's convolution module extracts the low-level feature maps of the training set images and inputs them into the decoder's PPM module. The encoder's detail branch and semantic branch perform two auxiliary tasks. The detail branch performs a boundary detection task, extracting defect boundary features from the training set images. The semantic branch performs a multi-label classification task, predicting the types of defects contained in the image, generating ground truth labels, and supervising the boundary features. The boundary features are used as high-level feature maps and input into the decoder. The ground truth labels are calculated as follows: the second-order gradient of the image is calculated using the Laplacian operator, and three boundary detail feature maps at different scales are obtained using three Laplacian convolutions of different lengths. Then, these three detail feature maps are upsampled to the original size of the input image, and then the three boundary detail features are concatenated and dynamically weighted through two-dimensional convolution to obtain the ground truth labels for the boundaries. The Laplacian operator uses an operator template for discrete calculation. The decoder's GCU module fuses the high-level feature map with the low-level feature map, and then uses bilinear interpolation upsampling and argmax to obtain the defect type detection result from the fused feature map; The auxiliary task uses a binary cross-entropy loss function to optimize network parameters and complete training. The trained object surface defect detection network is used to detect object surface defects, acquire and process object surface defect images, and identify the types of object surface defects.

2. The method as described in claim 1, characterized in that, The encoder's convolution module consists of six MobileNetV3 modules.

3. The method as described in claim 1, characterized in that, The encoder extracts high-level feature maps at 1 / 8 and 1 / 16 resolution of the image, and inputs the high-level feature maps into the decoder through skip connections of matrix element multiplication and addition; During the training phase, the high-level feature map at 1 / 8 resolution contains defect boundary features for detail branch detection; The GCU module includes two global context modules (GC), two upsampling modules (Upsample), and two group convolutional modules (GroupConv), in the order of GC+Upsample+GroupConv+Upsample+GroupConv+GC. The first Upsample fuses high-level feature maps at 1 / 16 resolution by element-wise addition, and the second Upsample fuses high-level feature maps at 1 / 8 resolution by element-wise multiplication.

4. The method as described in claim 1, characterized in that, The detail branch consists of convolution, batch normalization, ReLU function, convolution, and bilinear interpolation upsampling network; the semantic branch consists of convolution, batch normalization, ReLU function, convolution, global average pooling, and linear layer network.

5. A surface defect detection system for an object, characterized in that, The network includes an object surface defect detection network, comprising an encoder and a decoder; the encoder includes a convolutional module, a detail branch, and a semantic branch; the decoder includes a pyramid pooling module (PPM) and a global context upsampling module (GCU). After training, the object surface defect detection network is used to detect object surface defects, collect and process object surface defect images, and identify the types of object surface defects. The training steps for this object surface defect detection network include: The training set containing images of different defect types is input into the object surface defect detection network. The encoder's convolution module extracts the low-level feature maps of the training set images and inputs them into the decoder's PPM module. The encoder's detail branch and semantic branch perform two auxiliary tasks. The detail branch performs a boundary detection task, extracting defect boundary features from the training set images. The semantic branch performs a multi-label classification task, predicting the types of defects contained in the image, generating ground truth labels, and supervising the boundary features. The boundary features are used as high-level feature maps and input into the decoder. The ground truth labels are calculated as follows: the second-order gradient of the image is calculated using the Laplacian operator, and three boundary detail feature maps at different scales are obtained using three Laplacian convolutions of different lengths. Then, these three detail feature maps are upsampled to the original size of the input image, and then the three boundary detail features are concatenated and dynamically weighted through two-dimensional convolution to obtain the ground truth labels for the boundaries. The Laplacian operator uses an operator template for discrete calculation. The decoder's GCU module fuses the high-level feature map with the low-level feature map, and then uses bilinear interpolation upsampling and argmax to obtain the defect type detection result from the fused feature map; The auxiliary task employs a binary cross-entropy loss function, which is used to optimize network parameters and complete training.

6. The system as described in claim 5, characterized in that, The encoder's convolution module consists of six MobileNetV3 modules.

7. The system as described in claim 5, characterized in that, The encoder extracts high-level feature maps at 1 / 8 and 1 / 16 resolution of the image, and inputs the high-level feature maps into the decoder through skip connections of matrix element multiplication and addition; During the training phase, the high-level feature map at 1 / 8 resolution contains defect boundary features for detail branch detection; The GCU module includes two global context modules (GC), two upsampling modules (Upsample), and two group convolutional modules (GroupConv), in the order of GC+Upsample+GroupConv+Upsample+GroupConv+GC. The first Upsample fuses high-level feature maps at 1 / 16 resolution by element-wise addition, and the second Upsample fuses high-level feature maps at 1 / 8 resolution by element-wise multiplication.

8. The system as described in claim 5, characterized in that, The detail branch consists of convolution, batch normalization, ReLU function, convolution, and bilinear interpolation upsampling network; the semantic branch consists of convolution, batch normalization, ReLU function, convolution, global average pooling, and linear layer network.

Images