Multi-scale defect detection method based on deep learning network

Abstract

The invention discloses a multi-scale defect detection method based on a deep learning network, and belongs to the technical field of digital image processing. According to the method, a new defect detection MSE-YOLOv8 network is provided based on a YOLOv8 network, in a Backbone part, original Conv and C2f configuration is kept, and a local feature extraction layer and a global feature extraction layer are added; in the Neck part, an FPN + PAN layer is adopted, and a multi-scale feature fusion module is embedded; in the Head part, a semantic enhanced detection head is used, the design of a decoupling detection head continues to be used, and an improved space group enhancement module is adopted in a classification head to replace an original convolutional layer. Experimental results prove that compared with an existing detection method, the method has the advantages that the over-detection and missing detection conditions are optimized, and the detection precision is effectively improved.

Description

Technical Field

[0001] The present invention relates to a multi-scale defect detection method based on a deep learning network, and belongs to the technical field of digital image processing. Background Art

[0002] In the field of industrial inspection, the objects to be inspected may have defects of different sizes and shapes. These defects may be very subtle, such as tiny pits or glass spots; or they may be relatively large, such as surface contamination or oxide film residue. Multi-scale defect detection is a complex and challenging task. In-depth research on multi-scale defect detection technology is of great significance for product process improvement and yield control.

[0003] In traditional inspection methods, image difference is a common strategy that effectively locates defects by comparing the difference between the defect image and the reference grain image. In this process, how to construct a reference grain template image without interference noise and defects is a key issue.

[0004] Traditional algorithms are suitable for defects with small scale changes and simple structures, but they rely on artificially extracted features to detect defects, making it difficult to detect subdivided types of defects. When detecting defects with variable scales, the accuracy and stability are low. The development of deep learning technology provides a new solution to the problem of multi-scale defect detection. The wafer surface defect detection method based on YOLO shows good robustness. For example, "Zhang Heng, Cheng Cheng, Yuan Biao, et al. Research on chip defect detection method based on YOLOv5-EA-FPNs [J]. Journal of Electronic Measurement and Instrumentation, 2023, 37(05): 36-45" In order to solve the problems of large size span and similar features of wafer defects, a feature pyramid structure based on efficient focus learning is proposed to achieve multi-scale detection. "Xu Degang, Wang Zaiqing, Xing Kuijie, et al. Improved YOLOv6 remote sensing image target detection algorithm [J]. Computer Engineering and Applications, 2024, 60(03): 119-128" Based on YOLOv6, the coordinate attention and context enhancement modules are introduced to enhance the model's ability to understand the detailed information of multi-scale targets. Zhou Ying et al. designed a context aggregation module to fully extract the features of tiny defects, significantly improving the accuracy of YOLOv8 in small target detection. "Cui Kebin, Jiao Jingyi. Steel surface defect detection algorithm based on MCB-FAH-YOLOv8[J]. Journal of Graphics, 2024, 45(01):112-125" designed an adaptive feature fusion algorithm and added a detection head for small targets, so as to take into account both large-scale and small-scale target detection.

[0005] However, for the problem of defect detection with large scale changes, the existing mainstream detection algorithms still have certain limitations in dealing with such problems. In the feature extraction stage, the algorithm has limitations in processing details and global information, especially when dealing with defects of different sizes, the detection accuracy and generalization are reduced. Although the introduction of Transformer technology can improve performance, it will also lead to an imbalance in the dynamic balance between local and global information and cause a decrease in computational efficiency. In the feature fusion stage, the multi-scale characteristics of defects require that the fusion algorithm can effectively integrate feature maps at different resolutions. Traditional feature pyramid networks may introduce noise from features at different levels when dealing with multi-scale defect problems. These noises interfere with the accurate extraction of multi-scale target features, thereby reducing detection accuracy. Summary of the invention

[0006] In order to solve the problem of false detection and missed detection caused by large scale changes in the defect detection process, the present invention provides a multi-scale defect detection method based on a deep learning network. The technical solution is as follows:

[0007] The defect detection method of the present invention proposes a new defect detection MSE-YOLOv8 network based on the YOLOv8 network, trains the MSE-YOLOv8 network, and uses the trained MSE-YOLOv8 network to perform defect detection on the input image;

[0008] The MSE-YOLOv8 network improves the YOLOv8 network by: in the Backbone part, the original Conv and C2f configurations are maintained, and the local and global feature extraction layers LGLayer are added; in the Neck part, the FPN+PAN layer is used, and the multi-scale feature fusion module MSFF is embedded; in the Head part, the semantically enhanced detection head SE-Detect is used, the design of the decoupled detection head is used, and the improved spatial group enhancement module I-SGE is used in the classification head to replace the original convolution layer;

[0009] The global feature extraction layer LGLayer includes a Patch Embeding layer, a local feature extraction module NCB, a Flatten layer, a multi-head self-attention module MHSA and a Reshape layer connected in sequence. The calculation of the NCB module is expressed as:

[0010]

[0011] Among them, z l-1 represents the output of the previous layer, MHCA represents the multi-head convolutional attention structure, MLP represents the multi-layer perceptron, z l Represents the output of the NCB module.

[0012] Optionally, the MHCA module captures information from h parallel representation subspaces, and the calculation process is expressed as:

[0013]

[0014] where z=[z1,z2,...,z h ] indicates that the input feature z is divided into multiple forms in the channel dimension, W P represents the projection layer, CA represents the single-head convolution attention calculation, T m and T n are adjacent tokens in the input feature z, O represents the inner product operation, and W represents the trainable parameters.

[0015] Optionally, the I-SGE module includes: a Conv main branch and an SGE branch, and the outputs of the two branches are concatenated in the channel dimension and fused through convolution.

[0016] Optionally, the calculation process of the I-SGE module is expressed as:

[0017]

[0018] Among them, z i is the i-th feature after the input features are grouped, is the output of global average pooling, Norm means normalization according to the mean and variance of the feature, and Sigmoid is the activation layer.

[0019] The present invention provides a defect detection system, comprising:

[0020] A data acquisition module is configured to acquire an image to be detected;

[0021] A defect detection network is configured to train the MSE-YOLOv8 network and use the trained MSE-YOLOv8 network to perform defect detection on the input image;

[0022] The MSE-YOLOv8 network is improved based on the YOLOv8 network, including: in the Backbone part, the original Conv and C2f configurations are maintained, and the local and global feature extraction layers LGLayer are added; in the Neck part, the FPN+PAN layer is adopted, and the multi-scale feature fusion module MSFF is embedded; in the Head part, the semantically enhanced detection head SE-Detect is used, the design of the decoupled detection head is used, and the improved spatial group enhancement module I-SGE is adopted in the classification head to replace the original convolution layer;

[0023] The global feature extraction layer LGLayer includes a Patch Embeding layer, a local feature extraction module NCB, a Flatten layer, a multi-head self-attention module MHSA and a Reshape layer connected in sequence. The calculation of the NCB module is expressed as:

[0024]

[0025] Among them, z l-1 represents the output of the previous layer, MHCA represents the multi-head convolutional attention structure, MLP represents the multi-layer perceptron, z l Represents the output of the NCB module.

[0026] Optionally, the MHCA module captures information from h parallel representation subspaces, and the calculation process is expressed as:

[0027]

[0028] where z=[z1,z2,...,z h ] indicates that the input feature z is divided into multiple forms in the channel dimension, W P represents the projection layer, CA represents the single-head convolution attention calculation, T m and T n are adjacent tokens in the input feature z, O represents the inner product operation, and W represents the trainable parameters.

[0029] Optionally, the I-SGE module includes: a Conv main branch and an SGE branch, and the outputs of the two branches are concatenated in the channel dimension and fused through convolution.

[0030] Optionally, the calculation process of the I-SGE module is expressed as:

[0031]

[0032] Among them, z i is the i-th feature after the input features are grouped, is the output of global average pooling, Norm means normalization according to the mean and variance of the feature, and Sigmoid is the activation layer.

[0033] The present invention provides a defect detection device, comprising a memory and a processor;

[0034] The memory is used to store computer programs;

[0035] The processor is used to implement the defect detection method as described in any one of the above items when executing the computer program.

[0036] The present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the defect detection method as described in any one of the above items is implemented.

[0037] The beneficial effects of the present invention are:

[0038] Aiming at the problems of false detection and missed detection caused by large scale changes in the defect detection process, a multi-scale enhanced defect detection algorithm MSE-YOLOv8 is proposed. Firstly, local and global feature extraction layers are designed in the backbone network to capture rich local texture information and global structure information, and enhance the multi-scale feature extraction capability of the model; secondly, a multi-scale feature fusion module is designed to optimize the model's learning of the target's spatial position information and detail features when fusing feature maps from different levels; finally, the attention enhancement is performed on the semantic branch in the decoupled detection head to reduce the classification error of small targets and enhance the accuracy of multi-scale detection.

[0039] Experimental results show that compared with the existing detection methods, the present invention has optimized the over-detection and missed-detection situations and effectively improved the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 It is a structural diagram of the MSE-YOLOv8 network in the second embodiment of the present invention.

[0042] Figure 2 It is a structural diagram of the local and global feature extraction layers in the second embodiment of the present invention.

[0043] Figure 3 It is a structural diagram of the multi-scale feature fusion module in the second embodiment of the present invention.

[0044] Figure 4 It is a structural diagram of the improved space group enhancement module in the second embodiment of the present invention.

[0045] Figure 5 These are some examples of multi-scale defects, where (a) is a pit; (b) is a glass point; (c) is gold plating on the crystal corner; and (d) is residual oxide film.

[0046] Figure 6 It is a visualization diagram of the detection results of pits, glass spots and crystal corner gold plating defects at different scales by the present invention and the existing method.

[0047] Figure 7 It is a visualization diagram of the detection results of oxide film residue and crystal corner gold plating defects at different scales by the present invention and the existing method. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0049] Embodiment 1:

[0050] This embodiment provides a multi-scale defect detection method based on a deep learning network, proposes a new defect detection MSE-YOLOv8 network based on the YOLOv8 network, trains the MSE-YOLOv8 network, and uses the trained MSE-YOLOv8 network to perform defect detection on an input image.

[0051] The improvements of this embodiment to the YOLOv8 network mainly include the following points:

[0052] In the backbone network Backbone part, keep the original Conv and C2f configurations, and add the local and global feature extraction layer (Local and Global Layer, LGLayer).

[0053] In the Neck part, the FPN+PAN layer is used, and the Multi Scale Feature Fusion (MSFF) module is embedded.

[0054] In the Head part, the semantically enhanced detection head (Semantic-Enhanced Detect, SE-Detect) is used, the design of the decoupled detection head is adopted, and the improved spatial group enhancement module I-SGE (ImprovedSpatial Group-wise, I-SGE) is adopted in the classification head to replace the original convolutional layer.

[0055] The structure of the LGLayer module is as follows Figure 2 As shown in the figure, it includes the Patch Embedding layer, the local feature extraction module (Next Convolution Block, NCB), the Flatten layer, the Multi-Head Self-Attention module (Multi-Head Self-Attention, MHSA) and the Reshape layer connected in sequence. The output of the NCB module is Flattened into a one-dimensional sequence and then sent to the MHSA module, and then its calculation result is Reshaped into the output size of the NCB module.

[0056] Among them, the NCB module is essentially a module based on the convolution and attention framework, including a multi-head convolutional attention structure (Multi-Head Convolutional Attention, MHCA) and a multi-layer perceptron (MultilayerPerceptron, MLP). It not only has the locality of convolution, but also obtains the performance advantage of attention. The definition of the NCB module can be expressed as follows:

[0057]

[0058] Among them, z l-1 is the input of the previous layer, and z l They are the outputs of the MHCA module and the NCB module respectively.

[0059] The MHCA module is implemented based on the multi-head attention idea. It can simultaneously focus on information from different channels and perform effective local feature learning at different locations. The definition of the MHCA module can be summarized as follows:

[0060]

[0061] The MHCA module captures information from h parallel representation subspaces, z = [z1,z2,...,z h ] indicates that the input feature z is divided into multiple heads in the channel dimension; in order to promote information interaction between multiple heads, MHCA is equipped with a projection layer W P ; CA is single-head convolutional attention (Convolutional Attention, CA), T m and T n are adjacent tags in the input feature z, and O represents the inner product operation between the two. CA can learn the relationship between different tags in the local receptive field by iteratively optimizing the trainable parameter W, which is specifically achieved through group convolution and point-by-point convolution.

[0062] The structure of the multi-scale feature fusion module MSFF is as follows Figure 3 As shown in Figure 1, its main branch gradient module is an improved NBottleneck, which obtains richer gradient information while ensuring lightweight. The structure of MSFF and NBottleneck can be described as follows:

[0063]

[0064] Where z = [z1,z2,z nb ] represents the result of the input feature z being split by channel after the first Conv and Split. nb is the input of NBottleneck.

[0065] The structure of the space group enhancement module I-SGE is as follows Figure 4 As shown in the figure, it is mainly composed of the Conv main branch and the SGE branch. The outputs of the two branches are spliced ​​in the channel dimension and fused through a 1×1 convolution to form a semantically enhanced feature map. Its structure can be described as follows:

[0066]

[0067] Among them, z i is the result after the input features are divided into g groups, It is the output of global average pooling, representing the global features of the target. Norm refers to normalization based on the mean and variance of the features, and Sigmoid is the activation layer.

[0068] In order to verify the technical effect of the multi-scale defect detection method based on the MSE-YOLOv8 algorithm of the present invention, four types of defects, including pits, glass spots, gold plating at crystal corners and oxide film residues, are selected to form a multi-scale defect detection data set. Figure 5 The input size of the network training is set to 640×640. The dataset contains 1470 images in total, which are randomly divided into training set, validation set, and test set with 882, 294, and 294 images respectively. The detailed information of the dataset is shown in Table 1.

[0069] Table 1 Wafer multi-scale defect dataset

[0070]

[0071] The experimental environment is Windows 10 operating system, Intel Core i9-12900k CPU, GeForce RTX3090 GPU. The development environment is Python 3.8, Pytorch 1.10.0, CUDA 11.3, Pycharm IDE, Visual Studio 2022 IDE. The target detection model training parameters epoch is set to 200 rounds, Batch size is set to 16, the optimizer uses AdamW, the initial learning rate is 0.00125, the momentum is 0.9, and the decay is 0.0005. The model is trained from scratch without loading pre-trained weights.

[0072] In the systematic ablation experiment on the MSE-YOLOv8 network, three independent components were introduced into the network structure: LGLayer, MSFF, and SE-Detect, and their respective performance improvement effects are presented in Table 2.

[0073] Table 2 MSE-YOLOv8 ablation test results

[0074] Model LGLayer MSFF SE-Detect Params / M mAP@0.5 Baseline+ 11.17 0.890 Baseline+ √ 11.85 0.902 Baseline+ √ 13.15 0.900 Baseline+ √ 11.27 0.892 Baseline+ √ √ 13.83 0.906 Baseline+ √ √ √ 13.93 0.907

[0075] Specifically, adding LGLayer effectively improves the multi-scale feature extraction performance of the model, and the detection accuracy map@0.5 is increased by 1.2 percentage points, while the number of parameters increases by less than 1M. The MSFF structure plays a role in balancing the spatial details and semantic information in the feature maps at all levels, and the detection accuracy map@0.5 is improved by 1 percentage point, at the cost of an increase of nearly 2M parameters. After replacing the SE-Detect detection head, the semantically enhanced features enable the model to improve map@0.5 by 0.2 percentage points with an increase of only 0.1M parameters. When these components are applied jointly to the model, the performance improvement is even more significant. With the combined effect of the three modules, MSE-YOLOv8 achieves a significant increase of 1.7 percentage points in the map@0.5 indicator compared to the baseline network, while the number of model parameters only increases moderately by 2.76M.

[0076] The experiment selected 6 models including YOLOv5-s, PPYOLOE-s, YOLOv6-s, YOLOv8-s, YOLOv8-l, RT-DETR-L and MSE-YOLOv8 for comparison. The experimental data are shown in Table 3.

[0077] Table 3 Comparative test results

[0078]

[0079]

[0080] Analysis shows that the multi-scale defect detection method based on MSE-YOLOv8 proposed in this invention has the highest detection accuracy map@0.5, which is 1.7 percentage points higher than YOLOv8s, 1.2 percentage points higher than YOLOv8l, and 0.3 percentage points higher than RT-DETR-L, and the number of parameters is only 13.93M. In terms of detection speed, the method proposed in this invention is better than RT-DETR-L, lower than YOLOv8s and PPYOLOE-s, and is 19.1ms. The visualization of the comparison results is shown in the figure. Figure 6 , Figure 7 As shown in the figure, each row of images represents the recognition results of different methods, and the first column is the label image. The results show that the proposed method is superior to the comparison method in both over-detection and missed-detection in detecting multi-scale defects on the wafer surface.

[0081] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0082] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A defect detection method, characterized in that: The method proposes a new defect detection MSE-YOLOv8 network based on the YOLOv8 network, trains the MSE-YOLOv8 network, and uses the trained MSE-YOLOv8 network to perform defect detection on an input image; The MSE-YOLOv8 network improves the YOLOv8 network by: in the Backbone part, the original Conv and C2f configurations are maintained, and the local and global feature extraction layers LGLayer are added; in the Neck part, the FPN+PAN layer is used, and the multi-scale feature fusion module MSFF is embedded; in the Head part, the semantically enhanced detection head SE-Detect is used, the design of the decoupled detection head is used, and the improved spatial group enhancement module I-SGE is used in the classification head to replace the original convolution layer; The global feature extraction layer LGLayer includes a Patch Embeding layer, a local feature extraction module NCB, a Flatten layer, a multi-head self-attention module MHSA and a Reshape layer connected in sequence. The calculation of the NCB module is expressed as: Among them, z l-1 represents the output of the previous layer, MHCA represents the multi-head convolutional attention structure, MLP represents the multi-layer perceptron, z l Represents the output of the NCB module.

2. The defect detection method according to claim 1, characterized in that: The MHCA module captures information from h parallel representation subspaces, and the calculation process is expressed as: where z=[z1,z2,...,z h ] indicates that the input feature z is divided into multiple forms in the channel dimension, W P represents the projection layer, CA represents the single-head convolution attention calculation, T m and T n are adjacent tokens in the input feature z, O represents the inner product operation, and W represents the trainable parameters.

3. The defect detection method according to claim 1, characterized in that: The I-SGE module consists of a Conv main branch and an SGE branch. The outputs of the two branches are concatenated in the channel dimension and fused through convolution.

4. The defect detection method according to claim 3, characterized in that: The calculation process of the I-SGE module is expressed as: Among them, z i is the i-th feature after the input features are grouped, is the output of global average pooling, Norm means normalization according to the mean and variance of the feature, and Sigmoid is the activation layer.

5. A defect detection system, characterized in that: The system comprises: A data acquisition module is configured to acquire an image to be detected; A defect detection network is configured to train the MSE-YOLOv8 network and use the trained MSE-YOLOv8 network to perform defect detection on the input image; The MSE-YOLOv8 network is improved based on the YOLOv8 network, including: in the Backbone part, the original Conv and C2f configurations are maintained, and the local and global feature extraction layers LGLayer are added; in the Neck part, the FPN+PAN layer is adopted, and the multi-scale feature fusion module MSFF is embedded; in the Head part, the semantically enhanced detection head SE-Detect is used, the design of the decoupled detection head is used, and the improved spatial group enhancement module I-SGE is adopted in the classification head to replace the original convolution layer; The global feature extraction layer LGLayer includes a Patch Embeding layer, a local feature extraction module NCB, a Flatten layer, a multi-head self-attention module MHSA and a Reshape layer connected in sequence. The calculation of the NCB module is expressed as: Among them, z l-1 represents the output of the previous layer, MHCA represents the multi-head convolutional attention structure, MLP represents the multi-layer perceptron, z l Represents the output of the NCB module.

6. The defect detection system according to claim 5, characterized in that: The MHCA module captures information from h parallel representation subspaces, and the calculation process is expressed as: where z=[z1,z2,...,z h ] indicates that the input feature z is divided into multiple forms in the channel dimension, W P represents the projection layer, CA represents the single-head convolution attention calculation, T m and T n are adjacent tokens in the input feature z, O represents the inner product operation, and W represents the trainable parameters.

7. The defect detection system according to claim 5, characterized in that: The I-SGE module consists of a Conv main branch and an SGE branch. The outputs of the two branches are concatenated in the channel dimension and fused through convolution.

8. The defect detection system according to claim 7, characterized in that: The calculation process of the I-SGE module is expressed as: Among them, z i is the i-th feature after the input features are grouped, is the output of global average pooling, Norm means normalization according to the mean and variance of the feature, and Sigmoid is the activation layer.

9. A defect detection device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the defect detection method according to any one of claims 1 to 4 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the defect detection method according to any one of claims 1 to 4 is implemented.

Images