FPN wafer surface defect detection method based on parallel attention compensation mechanism
Through the parallel attention compensation mechanism and feature map pyramid network A-FPN, combined with K-means clustering, the anchor box size is improved, and the defect miss detection and false detection problems in wafer detection in complex background are solved, and the detection accuracy and coordination are improved. Small defect detection capability.
Patent Information
- Application Number
- CN202210588526.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-26
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-05-26
AI Technical Summary
Existing wafer detection technology is difficult to suppress background interference in complex backgrounds, resulting in the problems of missing and missed defects.
The FPN wafer surface defect detection method based on parallel attention compensation mechanism is adopted, feature extraction is performed through the ResNet network, feature map pyramid network A-FPN of fused attention is subdivided into the feature map, and the K-means clustering is improved. The size of the anchor box, combined with the channel and spatial attention mechanism, improves attention to defect characteristics and complex background suppression ability.
It effectively improves the detection ability of smaller defects and the detection accuracy of multi-scale defects, reduces the impact of complex background on the detection results, and avoids false detection and missed detection of small defects.
Smart Images

Figure CN114820580B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of wafer post-processing, and in particular to a FPN wafer surface defect detection method based on a parallel attention compensation mechanism. Background Art
[0002] During wafer defect detection, since the wafer surface has neatly patterned circuits, the neatly and regularly arranged circuit patterns will become a very complex background during visual inspection of the wafer. Therefore, during wafer inspection, defects appear as multi-scale features with various shapes. The above difficulties bring great challenges to the inspection task. Therefore, it is necessary to solve the problem of complex background interference during the inspection process and improve the accuracy of defect detection.
[0003] The main technology that needs to be solved this time is the existing wafer inspection technology, which is unable to suppress background image interference during the inspection process, resulting in problems such as defect omission and false detection. A multi-scale wafer surface defect detection method that integrates attention is proposed.
[0004] In 2018, Fang Xin et al. used the improved Faster Regions with CNN Features (Faster RCNN) method to detect wafer surface defects in the article "Wafer Defect Detection and Classification Algorithm Based on Convolutional Neural Network" (Computer Engineering, 2018, 44(08): 218-223). The method has good results for more significant defects, but due to the variety of wafer defects, the detection range of this method is limited.
[0005] In 2019, Zhang Yidong and others invented a method and system for detecting wafer defects (published as CN108831844B). This invention provides a wafer detection method and a wafer detection system, which uses infrared imaging to identify the defects of the wafer to be detected. However, the wafer surface has a complex background, so the defect detection is performed by template matching, and the detection accuracy is not high;
[0006] In 2022, Lian Haozhen et al. proposed a wafer defect detection method and system based on convolutional neural network (authorization number: CN112700406B). The invention mainly involves image preprocessing for detection. The wafer image captured by the image acquisition device is obtained, and weighted mean filtering is performed, and the processed wafer image is input into the defect detection model.
[0007] In summary, existing wafer defect detection methods involve image processing and detection of more significant characteristic defects, but do not involve the processing of complex background patterns, which have a greater impact on the detection results. Summary of the invention
[0008] The purpose of the present invention is to overcome the defects of the prior art and provide an FPN wafer surface defect detection method based on a parallel attention compensation mechanism, which can avoid the influence of complex background patterns on the defect detection effect, and at the same time prevent the missed detection and false detection of smaller targets or defects similar to background features.
[0009] The object of the present invention is achieved by: a FPN wafer surface defect detection method based on a parallel attention compensation mechanism, comprising the following steps:
[0010] Step 1) Extract wafer features based on ResNet network;
[0011] Step 2) The feature map pyramid network A-FPN that integrates attention subdivides the feature map;
[0012] Step 3) Improve the size generation of anchor boxes based on K-means clustering;
[0013] Step 4) obtain the candidate regions for classification and regression.
[0014] Furthermore, step 1) specifically includes using a ResNet-50 network to perform feature extraction to obtain four feature maps of different specifications, including low-level feature maps L2, L3 and high-level feature maps L4 and L5.
[0015] Furthermore, the step 2) specifically includes: setting a channel attention mechanism at the output position of the bottom-level feature maps L2 and L3, and obtaining fine feature maps C2 and C3 after channel attention fusion of L2 and L3; setting a spatial attention mechanism at the output position of the high-level feature map L4, and obtaining a fine feature map F'4 after spatial attention fusion of L4, and performing channel screening and area screening before inputting the feature map into the FPN network.
[0016] Furthermore, the execution process of the channel attention mechanism specifically includes:
[0017] First, average pooling and maximum pooling are performed on the feature map to extract two feature descriptions and And make the output of the feature map become 1×1×C / r, where r is the dimensionality reduction multiple;
[0018] Secondly, these two different features and They are sent to the middle shared network layer respectively, and then the corresponding elements of the two features are added, and then the channel attention map Mc is obtained by the sigmoid activation function, which is 1×1×C;
[0019] Finally, it is connected to the output of the original backbone network, as shown in Formula 1:
[0020] Mc(F)=σ(P MLP (P avg (F)+P MLP (P max (F))) (1)
[0021] Simplifying, we get:
[0022]
[0023] Among them, the shared network layer represents a multi-layer perceptron, which is represented by P in the formula. MLP It represents a neural network with two layers; P avg represents average pooling, P max represents maximum pooling, σ represents the sigmoid activation function, W0 and W1 represent the two-layer parameters of the multi-layer perceptron model, and the features between W0 and W1 are processed using ReLU as the activation function; finally, Mc(F) is multiplied by its input feature to obtain a fine feature map F' after channel attention adjustment; the obtained fine feature map F' is then input into the feature pyramid FPN for feature fusion.
[0024] Furthermore, the channel attention mechanism is set at the output position of the bottom feature maps L2 and L3, and L2 and L3 are fused through the channel attention to obtain the fine feature maps C2 and C3, which specifically includes:
[0025] After the 1×1 convolution, the feature map L2 is input into the channel attention mechanism. Assuming that the input feature map F is H×W×C, the attention map Mc2 is first obtained by the channel attention module l as 1×1×C. Mc2 is weighted for each channel of the input feature F, and then the refined feature F' is obtained; and then the feature map C2 is obtained by FPN feature fusion, as shown in Figure 3:
[0026]
[0027] in: Represents the multiplication of corresponding elements of the matrix, C represents the number of channels of the input feature, H and W represent the height and width of the feature map;
[0028] The feature map L3 is input into the channel attention mechanism after 1×1 convolution. Assuming that the input feature map F is H×W×C, the attention map Mc3 is first obtained by the channel attention module l as 1×1×C. Mc3 is weighted for each channel of the input feature F, and then the refined feature F' is obtained; and then the feature map C3 is obtained by FPN feature fusion, as shown in Figure 4:
[0029]
[0030] in: Represents the multiplication of corresponding elements of the matrix, C represents the number of channels of the input feature, and H and W represent the height and width of the feature map.
[0031] Furthermore, the spatial attention mechanism is set at the output position of the high-level feature map L4, and L4 is fused with spatial attention to obtain a fine feature map F'4, which specifically includes:
[0032] The fusion spatial attention mechanism is used for high-level feature maps. The feature map L4 is input to the spatial attention, and the feature map is obtained through average pooling and maximum pooling. is H×W×1 and The feature maps are H×W×1, and then a 7×7 convolution kernel is used after the feature maps are merged and a new spatial attention map Ms4 is obtained after the sigmiod activation function. The calculation formula of spatial attention is:
[0033]
[0034] Where: σ represents the sigmoid activation function, f 7×7 At the end of the convolution operation with a convolution kernel size of 7×7, Ms(F) is multiplied by its input features to obtain the final feature map F'4.
[0035] The present invention adopts the above technical solution, and compared with the prior art, the beneficial effects are as follows: the network model of Faster RCNN is adopted, and the feature graph pyramid network A-FPN (Attention Feature Pyramid Networks) is integrated to improve the detection ability of smaller defects and realize multi-scale defect detection, and also improve the model's attention to defect features and the ability to suppress complex background features.
[0036] In order to improve the model's attention to defect features and suppress the features of complex backgrounds, the feature map pyramid network A-FPN improved by fusion attention is used to subdivide the feature map. While improving the model's ability to detect defects of different scales in wafer images, it can also suppress the features of complex backgrounds.
[0037] Considering the characteristics of defects, the anchor box size is improved based on K-means clustering, which is closer to the anchor box of the actual target defect size in the data set, thereby improving the accuracy of position regression during defect detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 It is a schematic diagram of the overall process of the present invention.
[0039] Figure 2 This is a flow chart of FPN feature extraction based on fusion attention of the present invention.
[0040] Figure 3 Schematic diagram of the fusion channel attention of the present invention.
[0041] Figure 4 Schematic diagram of the fused spatial attention of the present invention.
[0042] Figure 5 It is a schematic diagram of feature fusion of the present invention.
[0043] Figure 6 This is a flowchart of the anchor box size generation based on K-means clustering improvement in the present invention. DETAILED DESCRIPTION
[0044] like Figure 1 The FPN wafer surface defect detection method based on the parallel attention compensation mechanism shown includes the following steps:
[0045] Step 1) extracting wafer features based on the ResNet network; Step 1) specifically includes using the ResNet-50 network to extract features, and obtaining four feature maps L2, L3, L4 and L5 of different specifications.
[0046] like Figure 2 As shown, step 2) the feature map pyramid network A-FPN that integrates attention subdivides the feature map; a channel attention mechanism is set at the output position of the bottom feature map L2 and L3, and L2 and L3 are fused with channel attention to obtain fine feature maps C2 and C3; a spatial attention mechanism is set at the output position of the high-level feature map L4, and L4 is fused with spatial attention to obtain a fine feature map F'4. Before the feature map is input into the FPN network, channel screening and region screening are performed first; larger weights are given to channels and regions that need attention, and smaller weights are given to unimportant channels and regions.
[0047] like Figure 3 As shown in Figure 2, the execution process of the channel attention mechanism specifically includes:
[0048] First, average pooling and maximum pooling are performed on the feature map to extract two feature descriptions and , and make the output of the feature map become 1×1×C / r, where r is the dimensionality reduction multiple;
[0049] Secondly, these two different features and They are sent to the middle shared network layer respectively, and then the corresponding elements of the two features are added, and then the channel attention map Mc is obtained by the sigmoid activation function, which is 1×1×C;
[0050] Finally, it is connected to the output of the original backbone network, as shown in Formula 6:
[0051] Mc(F)=σ(P MLP (P avg (F)+P MLP (P max (F))) (6)
[0052] Simplifying, we get:
[0053]
[0054] Among them, the shared network layer represents a multi-layer perceptron, which is represented by P in the formula. MLP It represents a neural network with two layers; P avg represents average pooling, P max represents maximum pooling, σ represents the sigmoid activation function, W0 and W1 represent the two-layer parameters of the multi-layer perceptron model, and the features between W0 and W1 are processed using ReLU as the activation function; finally, Mc(F) is multiplied by its input feature to obtain a fine feature map F' after channel attention adjustment; the obtained fine feature map F' is then input into the feature pyramid FPN for feature fusion.
[0055] A channel attention mechanism is set at the output position of the underlying feature maps L2 and L3. After L2 and L3 are fused through channel attention, fine feature maps C2 and C3 are obtained, which specifically include:
[0056] After the 1×1 convolution, the feature map L2 is input into the channel attention mechanism. Assuming that the input feature map F is H×W×C, the attention map Mc2 is first obtained by the channel attention module l as 1×1×C. Mc2 is weighted for each channel of the input feature F, and then the refined feature F' is obtained; and then the feature map C2 is obtained by FPN feature fusion, as shown in Figure 8:
[0057]
[0058] in: Represents the multiplication of corresponding elements of the matrix, C represents the number of channels of the input feature, H and W represent the height and width of the feature map;
[0059] After the 1×1 convolution, the feature map L3 is input into the channel attention mechanism. Assuming that the input feature map F is H×W×C, the attention map Mc3 is first obtained by the channel attention module l as 1×1×C. Mc3 is weighted for each channel of the input feature F, and then the refined feature F' is obtained; and then the feature map C3 is obtained by FPN feature fusion, as shown in Figure 9:
[0060]
[0061] in: Represents the multiplication of corresponding elements of the matrix, C represents the number of channels of the input feature, and H and W represent the height and width of the feature map.
[0062] like Figure 4 As shown in Figure 1, a spatial attention mechanism is set at the output position of the high-level feature map L4. After spatial attention fusion, L4 obtains a fine feature map F'4, which specifically includes:
[0063] The fusion spatial attention mechanism is used for high-level feature maps. The feature map L4 is input to the spatial attention, and the feature map is obtained through average pooling and maximum pooling. is H×W×1 and The feature maps are H×W×1, and then a 7×7 convolution kernel is used after the feature maps are merged and a new spatial attention map Ms4 is obtained after the sigmiod activation function. The calculation formula of spatial attention is:
[0064]
[0065] Where: σ represents the sigmoid activation function, f 7×7 At the end of the convolution operation with a convolution kernel size of 7×7, Ms(F) is multiplied by its input features to obtain the final feature map F'4.
[0066] There are two types of attention mechanisms: spatial attention and channel attention. For high-level features, only the semantic information of the defect is emphasized, and the location information of the defect is lost. Figure 5 As shown in the figure, the high-level feature map L4 is weighted and fused through the spatial attention mechanism to obtain a refined feature map F'4, which has the advantage of giving greater weight to areas with obvious defects. It is then fused with the C5 feature map that has been upsampled by 2 times to obtain the feature map C4, and the final feature map P4 is output through a 3×3 convolution.
[0067] Similarly, for the underlying features, there is a problem of emphasizing the position and contour information of the defect, so the weight fusion method through the inter-channel attention mechanism is adopted for the underlying feature map L2. The advantage is that the weight of the channels other than the position and contour information is reduced to obtain a refined feature map F'2. It is then fused with the C3 feature map that has been upsampled by 2 times to obtain the feature map C2, and finally the final feature map P2 is output through a 3×3 convolution.
[0068] Step 3) Improve the size generation of anchor boxes based on K-means clustering;
[0069] In order to generate an anchor box that is closer to the actual target defect size in the data set and improve the speed and accuracy of the algorithm in the defect detection process, k-means clustering is used to guide the setting of the anchor box size. Clustering is performed for the target defect size in a specific data set, that is, the intersection over union (IOU) of the manually marked ground-truth (true value) and the cluster center is calculated, and clustering is performed using the 1-IOU distance metric, which promotes RPN to generate an anchor box that better matches the shape of the actual defect, and further completes the location and detection of the defect.
[0070] like Figure 6 As shown, the anchor box flow chart is generated based on K-means clustering improvement.
[0071] 1) Given m cluster centers (W i ,H i ,), i∈{1,2…,m}, where W i , H i Indicates the width and height of the anchor box. Since its position is not fixed, there is no need to assign a value to the center point coordinates (x, y);
[0072] 2) The file containing the target location will be generated during the dataset annotation process, including (x j ,y j ,w j ,h j ,), j∈{1,2,…,N}, i.e., the coordinates of the ground-truth relative to the original image, where (x j ,y j ) is the coordinate of the center point of the box, (w j ,h j ) are width and height, N is the number of annotation boxes;
[0073] 3) Calculate the distance d between the annotation box in the data set and the m cluster center points, where:
[0074] d = 1-IOU[(x j ,y j ,w j ,h j ),(x j ,y j ,W i ,H i )], and divide the annotation box into the closest (W according to the size of d i ,H i )point;
[0075] 4) Until all the annotation boxes are assigned, calculate the final cluster center point for each cluster:
[0076]
[0077] Where: N i represents the number of annotation boxes in the i-th cluster. The whole formula is to find the average width and height of all the annotation boxes in the cluster;
[0078] 5) Repeat steps 3) and 4) until (W i ',H i ') changes very little. For the wafer defect dataset, k-means clustering is used to find the anchor box with a suitable aspect ratio.
[0079] Step 4) obtain the candidate regions for classification and regression.
[0080] The FPN wafer surface defect detection method based on the parallel attention mechanism provided by the present invention, compared with the current wafer detection method, combines the improved FPN with attention to suppress the complex circuit pattern background, avoiding the false detection and missed detection of minor defects; adopts the K-means clustering method to regenerate the anchor frame, and finds the anchor frame with a suitable aspect ratio for the wafer defect data set. Compared with the prior art, it shows that it can suppress the complex background, avoid the false detection and missed detection of minor defects, and has great market prospects and promotion value.
[0081] The present invention is not limited to the above-mentioned embodiments. On the basis of the technical solution disclosed in the present invention, technicians in this field can make some substitutions and deformations to some technical features therein according to the disclosed technical content without creative labor, and these substitutions and deformations are all within the protection scope of the present invention.
Claims
1. A FPN wafer surface defect detection method based on parallel attention compensation mechanism, characterized in that: The following steps are involved: Step 1) Extract wafer features based on ResNet network; Step 2) The feature map pyramid network A-FPN that integrates attention subdivides the feature map; Step 3) Improve the size generation of anchor boxes based on K-means clustering; Step 4) Obtain candidate regions for classification and regression; The step 2) specifically includes: setting a channel attention mechanism at the output position of the bottom feature maps L2 and L3, and obtaining fine feature maps C2 and C3 after L2 and L3 are fused through channel attention; setting a spatial attention mechanism at the output position of the high-level feature map L4, and obtaining a fine feature map F'4 after L4 is fused through spatial attention, and performing channel screening and region screening before inputting the feature map into the FPN network; The channel attention mechanism is set at the output position of the bottom feature maps L2 and L3, and L2 and L3 are fused through the channel attention to obtain the fine feature maps C2 and C3, which specifically include: After the 1×1 convolution, the feature map L2 is input into the channel attention mechanism. Assuming that the input feature map F is H×W×C, the attention map Mc2 is first obtained by the channel attention module l as 1×1×C. Mc2 is weighted for each channel of the input feature F, and then the refined feature F' is obtained; and then the feature map C2 is obtained by FPN feature fusion, as shown in Figure 3: in: Represents the multiplication of corresponding elements of the matrix, C represents the number of channels of the input feature, H and W represent the height and width of the feature map; The feature map L3 is input into the channel attention mechanism after 1×1 convolution. Assuming that the input feature map F is H×W×C, the attention map Mc3 is first obtained by the channel attention module l as 1×1×C. Mc3 is weighted for each channel of the input feature F, and then the refined feature F' is obtained; and then the feature map C3 is obtained by FPN feature fusion, as shown in Figure 4: in: Represents the multiplication of corresponding elements of the matrix, C represents the number of channels of the input feature, H and W represent the height and width of the feature map; The spatial attention mechanism is set at the output position of the high-level feature map L4, and L4 is fused with spatial attention to obtain a fine feature map F'4, which specifically includes: The fusion spatial attention mechanism is used for high-level feature maps. The feature map L4 is input to the spatial attention, and the feature map is obtained through average pooling and maximum pooling. is H×W×1 and The feature maps are H×W×1, and then a 7×7 convolution kernel is used after the feature maps are merged and a new spatial attention map Ms4 is obtained after the sigmiod activation function. The calculation formula of spatial attention is: Where: σ represents the sigmoid activation function, f 7×7 At the end of the convolution operation with a convolution kernel size of 7×7, Ms(F) is multiplied by its input features to obtain the final feature map F'4.
2. The FPN wafer surface defect detection method based on parallel attention compensation mechanism according to claim 1 is characterized in that: Step 1) specifically includes using the ResNet-50 network to perform feature extraction to obtain four feature maps of different specifications, including low-level feature maps L2, L3 and high-level feature maps L4 and L5.
3. The FPN wafer surface defect detection method based on parallel attention compensation mechanism according to claim 1 is characterized in that: The execution process of the channel attention mechanism specifically includes: First, average pooling and maximum pooling are performed on the feature map to extract two feature descriptions and And make the output of the feature map become 1×1×C / r, where r is the dimensionality reduction multiple; Secondly, these two different features and They are sent to the middle shared network layer respectively, and then the corresponding elements of the two features are added, and then the channel attention map Mc is obtained by the sigmoid activation function, which is 1×1×C; Finally, it is connected to the output of the original backbone network, as shown in Formula 1: Mc(F)=σ(P MLP (P avg (F)+P MLP (P max (F))) (1) Simplifying, we get: Among them, the shared network layer represents a multi-layer perceptron, which is represented by P in the formula. MLP It represents a neural network with two layers; P avg represents average pooling, P max represents maximum pooling, σ represents the sigmoid activation function, W0 and W1 represent the two-layer parameters of the multi-layer perceptron model, and the features between W0 and W1 are processed using ReLU as the activation function; finally, Mc(F) is multiplied by its input feature to obtain a fine feature map F' after channel attention adjustment; the obtained fine feature map F' is then input into the feature pyramid FPN for feature fusion.
