Multi-scale defect detection method based on deep learning network

Abstract

The present invention discloses a multi-scale defect detection method based on a deep learning network, belonging to the field of digital image processing technology. The method proposes a new defect detection MSE-YOLOv8 network based on the YOLOv8 network. In the Backbone part, the original Conv and C2f configurations are maintained, and local and global feature extraction layers are added. In the Neck part, the FPN+PAN layer is adopted, and a multi-scale feature fusion module is embedded. In the Head part, a semantically enhanced detection head is used, the design of the decoupled detection head is adopted, and an improved spatial group enhancement module is adopted in the classification head, replacing the original convolution layer. Experimental results show that compared with existing detection methods, the present invention has optimized over-detection and missed detection, effectively improving detection accuracy.

Description

Technical Field

[0001] The present invention relates to a multi-scale defect detection method based on a deep learning network, and belongs to the technical field of digital image processing. Background Art

[0002] In the field of industrial inspection, objects under inspection may exhibit defects of varying sizes and shapes. These defects can be extremely subtle, such as tiny pits or glass spots, or relatively large, such as surface contamination or residual oxide films. Multi-scale defect detection is a complex and challenging task, and in-depth research on multi-scale defect detection technology is of great significance for product process improvement and yield control.

[0003] Image differencing is a common strategy used in traditional inspection methods to effectively locate defects by comparing defect images with reference grain images. However, constructing a reference grain template image that is free of noise and defects is a key issue in this process.

[0004] Traditional algorithms are suitable for defects with small scale changes and simple structures, but they rely on manually extracted features to detect defects, making it difficult to detect subdivided types of defects. When detecting defects with varying scales, the accuracy and stability are low. The development of deep learning technology has provided a new solution to the problem of multi-scale defect detection. The wafer surface defect detection method based on YOLO shows good robustness. For example, "Zhang Heng, Cheng Cheng, Yuan Biao, et al. Research on chip defect detection method based on YOLOv5-EA-FPNs [J]. Journal of Electronic Measurement and Instrumentation, 2023, 37(05): 36-45" proposed a feature pyramid structure based on efficient focus learning to achieve multi-scale detection in response to the problems of large size span and feature similarity of wafer defects. "Xu Degang, Wang Zaiqing, Xing Kuijie, et al. Improved remote sensing image target detection algorithm of YOLOv6 [J]. Computer Engineering and Applications, 2024, 60(03): 119-128" introduced coordinate attention and context enhancement modules based on YOLOv6 to enhance the model's ability to understand multi-scale target detail information. Zhou Ying et al. designed a context aggregation module to fully extract the features of small defects, significantly improving the accuracy of YOLOv8 in small target detection. "Cui Kebin, Jiao Jingyi. Steel surface defect detection algorithm based on MCB-FAH-YOLOv8[J]. Journal of Graphics, 2024, 45(01):112-125" designed an adaptive feature fusion algorithm and added a detection head for small targets, thereby taking into account both large-scale and small-scale target detection.

[0005] However, existing mainstream detection algorithms still have certain limitations in dealing with defects with large scale variations. In the feature extraction stage, the algorithm has limitations in processing details and global information, especially when dealing with defects of different sizes, which reduces detection accuracy and generalization. Although the introduction of Transformer technology can improve performance, it will also lead to an imbalance in the dynamic balance between local and global information and cause a decrease in computational efficiency. In the feature fusion stage, the multi-scale characteristics of defects require the fusion algorithm to effectively integrate feature maps at different resolutions. Traditional feature pyramid networks may introduce noise from features at different levels when dealing with multi-scale defect problems. These noises interfere with the accurate extraction of multi-scale target features, thereby reducing detection accuracy. Summary of the Invention

[0006] In order to solve the problems of false detection and missed detection caused by large scale changes during defect detection, the present invention provides a multi-scale defect detection method based on a deep learning network. The technical solution is as follows:

[0007] The defect detection method of the present invention proposes a new defect detection MSE-YOLOv8 network based on the YOLOv8 network, trains the MSE-YOLOv8 network, and uses the trained MSE-YOLOv8 network to perform defect detection on the input image;

[0008] The MSE-YOLOv8 network improves the YOLOv8 network by: maintaining the original Conv and C2f configurations in the Backbone part and adding the local and global feature extraction layer LGLayer; adopting the FPN+PAN layer in the Neck part and embedding the multi-scale feature fusion module MSFF; using the semantically enhanced detection head SE-Detect in the Head part, continuing the design of the decoupled detection head, and adopting the improved spatial group enhancement module I-SGE in the classification head to replace the original convolutional layer;

[0009] The global feature extraction layer LGLayer includes a Patch Embeding layer, a local feature extraction module NCB, a Flatten layer, a multi-head self-attention module MHSA, and a Reshape layer connected in sequence. The calculation of the NCB module is expressed as:

[0010]

[0011] Among them, z l-1 Represents the output of the previous layer, MHCA represents the multi-head convolutional attention structure, MLP represents the multi-layer perceptron, z l Represents the output of the NCB module.

[0012] Optionally, the MHCA module captures information from h parallel representation subspaces. The calculation process is expressed as:

[0013]

[0014] Where z=[z1,z2,...,z h ] indicates that the input feature z is divided into multiple forms in the channel dimension, W P represents the projection layer, CA represents the single-head convolution attention calculation, T m and T n are adjacent tokens in the input feature z, O represents the inner product operation, and W represents the trainable parameters.

[0015] Optionally, the I-SGE module includes: a Conv main branch and an SGE branch, where the outputs of the two branches are concatenated in the channel dimension and fused through convolution.

[0016] Optionally, the calculation process of the I-SGE module is expressed as:

[0017]

[0018] Among them, z i is the i-th feature after the input features are grouped, is the output of global average pooling, Norm means normalization according to the mean and variance of the feature, and Sigmoid is the activation layer.

[0019] The present invention provides a defect detection system, comprising:

[0020] A data acquisition module is configured to acquire an image to be detected;

[0021] A defect detection network is configured to train the MSE-YOLOv8 network and use the trained MSE-YOLOv8 network to perform defect detection on input images;

[0022] The MSE-YOLOv8 network is improved based on the YOLOv8 network. The following steps are taken: in the backbone part, the original Conv and C2f configurations are maintained, and the local and global feature extraction layers LGLayer are added; in the neck part, the FPN+PAN layer is adopted, and the multi-scale feature fusion module MSFF is embedded; in the head part, the semantically enhanced detection head SE-Detect is used, the design of the decoupled detection head is retained, and the improved spatial group enhancement module I-SGE is adopted in the classification head to replace the original convolutional layer;

[0023] The global feature extraction layer LGLayer includes a Patch Embeding layer, a local feature extraction module NCB, a Flatten layer, a multi-head self-attention module MHSA, and a Reshape layer connected in sequence. The calculation of the NCB module is expressed as:

[0024]

[0025] Among them, z l-1 Represents the output of the previous layer, MHCA represents the multi-head convolutional attention structure, MLP represents the multi-layer perceptron, z l Represents the output of the NCB module.

[0026] Optionally, the MHCA module captures information from h parallel representation subspaces. The calculation process is expressed as:

[0027]

[0028] Where z=[z1,z2,...,z h ] indicates that the input feature z is divided into multiple forms in the channel dimension, W P represents the projection layer, CA represents the single-head convolution attention calculation, T m and T n are adjacent tokens in the input feature z, O represents the inner product operation, and W represents the trainable parameters.

[0029] Optionally, the I-SGE module includes: a Conv main branch and an SGE branch, where the outputs of the two branches are concatenated in the channel dimension and fused through convolution.

[0030] Optionally, the calculation process of the I-SGE module is expressed as:

[0031]

[0032] Among them, z i is the i-th feature after the input features are grouped, is the output of global average pooling, Norm means normalization according to the mean and variance of the feature, and Sigmoid is the activation layer.

[0033] The present invention provides a defect detection device, comprising a memory and a processor;

[0034] The memory is used to store computer programs;

[0035] The processor is configured to implement any one of the above defect detection methods when executing the computer program.

[0036] The present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the defect detection method as described in any one of the above items is implemented.

[0037] The beneficial effects of the present invention are:

[0038] To address the problem of false detection and missed detection caused by large scale variations during defect detection, a multi-scale enhanced defect detection algorithm, MSE-YOLOv8, is proposed. First, local and global feature extraction layers are designed within the backbone network to capture rich local texture information and global structural information, enhancing the model's multi-scale feature extraction capabilities. Second, a multi-scale feature fusion module is designed to optimize the model's learning of target spatial position information and detailed features by fusing feature maps from different layers. Finally, attention enhancement is performed on the semantic branch of the decoupled detection head to reduce classification errors for small objects and enhance the accuracy of multi-scale detection.

[0039] Experimental results show that compared with existing detection methods, the present invention has optimized over-detection and missed detection situations and effectively improved detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 4 is a structural diagram of the MSE-YOLOv8 network in the second embodiment of the present invention.

[0042] Figure 2 4 is a structural diagram of the local and global feature extraction layers in the second embodiment of the present invention.

[0043] Figure 3 4 is a structural diagram of the multi-scale feature fusion module in the second embodiment of the present invention.

[0044] Figure 4 It is a structural diagram of the improved space group enhancement module in the second embodiment of the present invention.

[0045] Figure 5 These are some examples of multi-scale defects, where (a) is a pit; (b) is a glass point; (c) is gold plating on the crystal corner; and (d) is residual oxide film.

[0046] Figure 6 This is a visualization of the detection results of pits, glass spots, and crystal corner gold plating defects at different scales using the present invention and existing methods.

[0047] Figure 7 This is a visualization of the detection results of oxide film residue and crystal corner gold plating defects at different scales using the present invention and existing methods. DETAILED DESCRIPTION

[0048] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0049] Example 1:

[0050] This embodiment provides a multi-scale defect detection method based on a deep learning network. A new defect detection MSE-YOLOv8 network is proposed based on the YOLOv8 network. The MSE-YOLOv8 network is trained and the trained MSE-YOLOv8 network is used to perform defect detection on an input image.

[0051] This embodiment improves the YOLOv8 network in the following aspects:

[0052] In the backbone network, the original Conv and C2f configurations are maintained, and local and global feature extraction layers (LGLayer) are added.

[0053] In the Neck part, the FPN+PAN layer is used and the Multi Scale Feature Fusion (MSFF) module is embedded.

[0054] In the head part, a semantically enhanced detection head (SE-Detect) is used, which follows the design of the decoupled detection head. In the classification head, an improved spatial group enhancement module I-SGE (Improved Spatial Group-wise, I-SGE) is adopted to replace the original convolutional layer.

[0055] The structure of the LGLayer module is as follows Figure 2 As shown in the figure, it includes the Patch Embeding layer, the local feature extraction module (Next Convolution Block, NCB), the Flatten layer, the Multi-Head Self-Attention module (Multi-Head Self-Attention, MHSA) and the Reshape layer connected in sequence. The output of the NCB module is Flattened into a one-dimensional sequence and then sent to the MHSA module, and then its calculation result is reshaped into the output size of the NCB module.

[0056] The NCB module is essentially a module based on the convolution and attention framework, including a multi-head convolutional attention structure (MHCA) and a multi-layer perceptron (MLP). It not only has the locality of convolution but also obtains the performance advantages of attention. The definition of the NCB module can be expressed as follows:

[0057]

[0058] Among them, z l-1 is the input of the previous layer, and z l are the outputs of the MHCA module and the NCB module respectively.

[0059] The MHCA module is implemented based on the multi-head attention concept. It can simultaneously focus on information from different channels and perform effective local feature learning at different locations. The definition of the MHCA module can be summarized as follows:

[0060]

[0061] The MHCA module captures information from h parallel representation subspaces, z = [z1,z2,...,z h ] indicates that the input feature z is divided into multiple heads in the channel dimension; in order to promote information interaction between multiple heads, MHCA is equipped with a projection layer W P ;CA is single-head convolutional attention (ConvolutionalAttention, CA), T m and T n are adjacent tags in the input feature z, and O represents the inner product operation between the two. CA can learn the relationship between different tags in the local receptive field by iteratively optimizing the trainable parameter W, which is specifically achieved through group convolution and point-by-point convolution.

[0062] The structure of the multi-scale feature fusion module MSFF is as follows Figure 3 As shown, its main branch gradient module is an improved NBottleneck, which obtains richer gradient information while ensuring lightweight. The structure of MSFF and NBottleneck can be described as follows:

[0063]

[0064] Where z=[z1,z2,z nb ] represents the result of the input feature z being split by channel after the first Conv and Split, z nb is the input of NBottleneck.

[0065] The structure of the space group enhancement module I-SGE is as follows Figure 4 As shown in the figure, it mainly consists of the Conv main branch and the SGE branch. The outputs of the two branches are spliced ​​in the channel dimension and fused through a 1×1 convolution to form a semantically enhanced feature map. Its structure can be described as follows:

[0066]

[0067] Among them, z i is the result after the input features are divided into g groups, It is the output of global average pooling and represents the global characteristics of the target. Norm refers to normalization based on the mean and variance of the features, and Sigmoid is the activation layer.

[0068] In order to verify the technical effect of the multi-scale defect detection method based on the MSE-YOLOv8 algorithm of the present invention, a total of four types of defects, including pits, glass points, crystal corner gold plating and oxide film residues, are selected to form a multi-scale defect detection data set. Figure 5 The input size for network training is set to 640×640. The dataset contains 1470 images in total, randomly divided into training, validation, and test sets of 882, 294, and 294 images, respectively. Details of the dataset are shown in Table 1.

[0069] Table 1 Wafer multi-scale defect dataset

[0070]

[0071] The experimental environment uses a Windows 10 operating system, an Intel Core i9-12900k CPU, and a GeForce RTX3090 GPU. The development environment uses Python 3.8, PyTorch 1.10.0, CUDA 11.3, PyCharm IDE, and Visual Studio 2022 IDE. The object detection model training parameters are set to 200 epochs, a batch size of 16, and the AdamW optimizer with an initial learning rate of 0.00125, a momentum of 0.9, and a decay of 0.0005. The model is trained from scratch without loading pretrained weights.

[0072] In the systematic ablation experiment on the MSE-YOLOv8 network, three independent components were introduced into the network structure: LGLayer, MSFF, and SE-Detect, and their respective performance improvements are presented in Table 2.

[0073] Table 2 MSE-YOLOv8 ablation test results

[0074] Model LGLayer MSFF SE-Detect Params / M mAP@0.5 Baseline+ 11.17 0.890 Baseline+ √ 11.85 0.902 Baseline+ √ 13.15 0.900 Baseline+ √ 11.27 0.892 Baseline+ √ √ 13.83 0.906 Baseline+ √ √ √ 13.93 0.907

[0075] Specifically, the addition of the LGLayer effectively improves the model's multi-scale feature extraction performance, increasing detection accuracy (map@0.5) by 1.2 percentage points while increasing the number of parameters by less than 1M. The MSFF structure balances the spatial detail and semantic information in feature maps at all levels, improving detection accuracy (map@0.5) by 1 percentage point at the expense of a nearly 2M increase in parameters. After replacing the SE-Detect detection head, the semantically enhanced features improve the model's map@0.5 by 0.2 percentage points, while increasing the number of parameters by only 0.1M. When these components are combined in the model, the performance improvement is even more significant. With the combined effect of these three modules, MSE-YOLOv8 achieves a significant 1.7 percentage point increase in map@0.5 compared to the baseline network, while maintaining a modest increase in model parameters to 2.76M.

[0076] The experiment selected 6 models including YOLOv5-s, PPYOLOE-s, YOLOv6-s, YOLOv8-s, YOLOv8-l, RT-DETR-L and MSE-YOLOv8 for comparison. The experimental data are shown in Table 3.

[0077] Table 3 Comparative test results

[0078]

[0079]

[0080] Analysis shows that the multi-scale defect detection method based on MSE-YOLOv8 proposed in this paper has the highest detection accuracy map@0.5, which is 1.7 percentage points higher than YOLOv8s, 1.2 percentage points higher than YOLOv8l, and 0.3 percentage points higher than RT-DETR-L, and the number of parameters is only 13.93M. In terms of detection speed, the method proposed in this paper is better than RT-DETR-L, lower than YOLOv8s and PPYOLOE-s, and is 19.1ms. The visualization of the comparison results is shown in the figure. Figure 6 、 Figure 7 As shown in the figure, each row of images represents the recognition results of different methods, and the first column is the label image. The results show that the proposed method is superior to the comparison method in both over-detection and missed detection in detecting multi-scale defects on the wafer surface.

[0081] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0082] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A defect detection method, characterized in that: The method proposes a new defect detection MSE-YOLOv8 network based on the YOLOv8 network, trains the MSE-YOLOv8 network, and uses the trained MSE-YOLOv8 network to perform defect detection on the input image; The MSE-YOLOv8 network improves the YOLOv8 network by: maintaining the original Conv and C2f configurations in the Backbone part and adding the local and global feature extraction layer LGLayer; adopting the FPN+PAN layer in the Neck part and embedding the multi-scale feature fusion module MSFF; using the semantically enhanced detection head SE-Detect in the Head part, continuing the design of the decoupled detection head, and adopting the improved spatial group enhancement module I-SGE in the classification head to replace the original convolutional layer; The local and global feature extraction layer LGLayer includes a Patch Embeding layer, a local feature extraction module NCB, a Flatten layer, a multi-head self-attention module MHSA and a Reshape layer connected in sequence. The calculation of the NCB module is expressed as: Among them, z l-1 Represents the output of the previous layer, MHCA represents the multi-head convolutional attention structure, MLP represents the multi-layer perceptron, z l Represents the output of the NCB module, Represents the output of the MHCA module; The I-SGE module consists of a Conv main branch and an SGE branch. The outputs of the two branches are concatenated in the channel dimension and fused through convolution.

2. The defect detection method according to claim 1, characterized in that: The MHCA module captures information from h parallel representation subspaces, and the calculation process is expressed as: Where z=[z1,z2,...,z h ] indicates that the input feature z is divided into multiple forms in the channel dimension, W P represents the projection layer, CA represents the single-head convolution attention calculation, T m and T n are adjacent tokens in the input feature z, O represents the inner product operation, and W represents the trainable parameters.

3. The defect detection method according to claim 2, characterized in that: The calculation process of the I-SGE module is expressed as: Among them, z i is the i-th feature after the input features are grouped, is the output of global average pooling, Norm means normalization according to the mean and variance of the feature, and Sigmoid is the activation layer.

4. A defect detection system, characterized in that: The system comprises: A data acquisition module is configured to acquire an image to be detected; A defect detection network is configured to train the MSE-YOLOv8 network and use the trained MSE-YOLOv8 network to perform defect detection on input images; The MSE-YOLOv8 network is improved based on the YOLOv8 network. The following steps are taken: in the backbone part, the original Conv and C2f configurations are maintained, and the local and global feature extraction layers LGLayer are added; in the neck part, the FPN+PAN layer is adopted, and the multi-scale feature fusion module MSFF is embedded; in the head part, the semantically enhanced detection head SE-Detect is used, the design of the decoupled detection head is retained, and the improved spatial group enhancement module I-SGE is adopted in the classification head to replace the original convolutional layer; The local and global feature extraction layer LGLayer includes a Patch Embeding layer, a local feature extraction module NCB, a Flatten layer, a multi-head self-attention module MHSA and a Reshape layer connected in sequence. The calculation of the NCB module is expressed as: Among them, z l-1 Represents the output of the previous layer, MHCA represents the multi-head convolutional attention structure, MLP represents the multi-layer perceptron, z l Represents the output of the NCB module, Represents the output of the MHCA module; The I-SGE module consists of a Conv main branch and an SGE branch. The outputs of the two branches are concatenated in the channel dimension and fused through convolution.

5. The defect detection system according to claim 4, characterized in that: The MHCA module captures information from h parallel representation subspaces, and the calculation process is expressed as: Where z=[z1,z2,...,z h ] indicates that the input feature z is divided into multiple forms in the channel dimension, W P represents the projection layer, CA represents the single-head convolution attention calculation, T m and T n are adjacent tokens in the input feature z, O represents the inner product operation, and W represents the trainable parameters.

6. The defect detection system according to claim 5, characterized in that: The calculation process of the I-SGE module is expressed as: Among them, z i is the i-th feature after the input features are grouped, z i is the output of global average pooling, Norm means normalization according to the mean and variance of the feature, and Sigmoid is the activation layer.

7. A defect detection device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the defect detection method according to any one of claims 1 to 3 when executing the computer program.

8. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the defect detection method according to any one of claims 1 to 3 is implemented.

Images