Wafer object segmentation method based on boundary-aware SegFormer

Abstract

The present invention discloses a wafer target segmentation method based on boundary-aware SegFormer, and belongs to the field of digital image processing technology. The present invention proposes an adaptive seed search strategy, uses the correlation between seed position and target positioning accuracy to construct a seed distribution heat map, and adaptively searches for ideal seed targets under the guidance of the heat map to achieve high-precision global segmentation of array targets; the boundary-aware SegFormer network uses recursive gated convolution to emphasize the long-distance and high-order spatial interaction of features, and the improved gated residual boundary refinement module can learn richer edge information. At the same time, a hybrid loss function is introduced to strengthen the supervision of pixels inside and at the edge of the region, guiding the network to better learn the target edge features and improve the boundary segmentation accuracy. The segmentation method of the present invention can completely and accurately segment targets in high-resolution array target images with uneven background, defect pollution, and low edge contrast, and has high real-time performance.

Description

Technical Field

[0001] The invention relates to a wafer target segmentation method based on boundary perception SegFormer, and belongs to the technical field of digital image processing. Background Art

[0002] Array targets are usually composed of multiple similar sub-targets arranged in a regular pattern, such as grains on a wafer and microarrays on a biochip, which are widely used in semiconductor manufacturing, industrial inspection, and medicine. The precise segmentation of targets directly affects the positioning accuracy, sorting performance, and defect detection accuracy. However, array targets are affected by non-uniform background, defect interference, and weak edges, resulting in low segmentation accuracy and incomplete edges. Therefore, studying array target segmentation algorithms in complex scenes has important research significance and value in the fields of target detection, defect detection, and image analysis.

[0003] Traditional array target image segmentation methods use manually selected features to segment targets. Although they have high accuracy in segmenting high-contrast targets, they have low accuracy when processing targets in complex scenes, are prone to losing edge information, and have poor stability. The article "Chen X, Chen J, Han X, et al. A Light-Weighted CNN Model for Wafer Structural Defect Detection [J]. IEEE Access, 2020, 8: 24006-18" uses the Otsu method and morphological methods to extract the grayscale features of array targets to segment targets, and the article "Xu S, Cheng Z, Gao Y, et al. Visual wafer dies counting using geometrical characteristics [J]. IET Image Processing, 2014, 8 (5): 280-8" fits the straight line features in the array based on the RANSAC method, and achieves target segmentation after gridding the array. However, under the interference of non-uniform background and defects, the segmentation accuracy of these methods is greatly reduced, and the edge segmentation is incomplete. To address this problem, some scholars have proposed a fixed-size Mask segmentation method, such as "Yang Dongxu. Research on Chip Sorting System Based on Machine Vision [D]. Beijing: Electronic Science Research Institute of China Electronics Technology Group Corporation, 2022." The document uses a template matching method to locate the target center of mass, and then generates a fixed-size rectangular Mask to indirectly segment the target. Although this type of method improves segmentation accuracy and stability, it is sensitive to the choice of template, has low accuracy when segmenting defective targets with low edge contrast, and there is also a redundant background problem when the target size does not match the template. In addition, without downsampling, the above traditional methods are slow when segmenting high-resolution images.

[0004] In recent years, deep networks have attracted much attention from researchers because of their ability to capture richer features and multi-scale information. Compared with traditional manually constructed features, the features extracted by deep networks have significant performance. Therefore, array target segmentation methods based on deep learning have been widely studied and can be divided into methods based on detection networks and segmentation networks. Some methods use detection networks to locate and then indirectly segment targets, such as YOLOv5 and WPCCNN. Compared with traditional methods, they effectively reduce the interference of background and defects and have higher segmentation accuracy. Among them, in the document "Peng JQ, Yang ZX, Ying H, et al. Bougie location algorithm of lattice biochip based on YOLOv5[J]. Journal of Electronic Imaging, 2022, 31:063006", Peng et al. proposed a fixed seed inference method based on YOLOv5, which can infer the target position that other networks have not detected by using the array distribution characteristics. However, when used for high-resolution images with hundreds of millions of pixels, it has a large inference error and the segmented target edges are incomplete. Different from the positioning method of rectangular boxes in detection networks, image segmentation networks achieve end-to-end semantic segmentation by predicting the semantic information of each pixel, making the target edge segmentation more continuous and the segmentation accuracy higher. In order to improve the segmentation accuracy of edge areas, researchers have proposed optimization methods from the aspects of loss function and network structure. For example, the residual refinement module (RRM) that emphasizes the quality of boundary segmentation, the interactive two-stream decoder, and the active boundary loss for boundary refinement. Among them, the RRM proposed in the document "Qin X, Zhang Z, Huang C, et al. BASNet: Boundary-Aware Salient Object Detection [J]. 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019: 7471-81" uses the U-shaped architecture to capture rich semantic information to achieve boundary refinement, but it cannot well preserve the underlying position information of long distances, and the boundary segmentation accuracy still has room for improvement. In addition, global segmentation accuracy and speed are also important performance indicators of segmentation methods, and are also hot topics for scholars to study. Some methods based on convolutional neural networks and self-attention are constantly proposed.Among them, "Xie E, Wang W, Yu Z, et al. SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers [J]. ArXiv, 2021, abs / 2105.15203." proposed a self-attention-based SegFormer network, which uses a hierarchical Transformer encoder to generate high-resolution fine features and low-resolution coarse features, and then aggregates these features through an MLP decoder to achieve high segmentation accuracy and speed. Compared with traditional methods, the above deep learning methods have higher segmentation accuracy and stability, but the segmentation of target edge areas is still a challenging problem under the interference of non-uniform background and defects. Summary of the invention

[0005] In order to improve the accuracy of array target segmentation under the interference of non-uniform background and defects, the present invention provides a wafer target segmentation method based on boundary-aware SegFormer, and the technical solution is as follows:

[0006] A first object of the present invention is to provide a wafer array target image segmentation method, comprising:

[0007] Step 1: Collect the array target image, divide the array target image into m regions, and i Parallel processing, i = 0, 1, ..., m;

[0008] Step 2: Adopting an adaptive seed search strategy to search for an ideal seed target in the region, the adaptive seed search strategy comprising: constructing a seed distribution heat map using the correlation between the seed position and the target positioning accuracy, and adaptively searching for the ideal seed target under the guidance of the heat map;

[0009] Step 3: Taking each ideal seed target as the starting point, predict the centroid position and rectangular Mask of all targets to achieve region R i Global segmentation of internal objects;

[0010] Step 4: Use the boundary-aware SegFormer network to accurately segment the global segmentation result to obtain the segmentation result of the array target image.

[0011] Optionally, the adaptive seed search strategy includes:

[0012] Step 21: Locate area R first iThe rectangular boundary of the image is obtained, and four strip regions of interest ROIs parallel to the boundary are taken at the image boundary. Then, the target centroid sets in the four ROIs are obtained by multi-target template matching, and the centroids close to the image boundary are taken to form the array area R i Finally, it is divided into 4 sub-rectangular regions, with overlaps between regions to avoid losing the target at the region boundary;

[0013] Step 22: Generate a seed distribution heat map using the correlation between the seed position and the target positioning accuracy, select the seed with the highest weight as the initial seed, and output its actual position in the image;

[0014] Step 23: Generate a ROI twice the size of the template based on the actual position of the initial seed in the image, and then locate the target based on template matching. If the normalized correlation coefficient is less than the set threshold, search for the next candidate seed target based on the heat map until four ideal seed targets are successfully located.

[0015] Optionally, step 3 includes: taking each ideal seed target as a starting point, reasoning along the horizontal and vertical directions, and predicting the centroid positions and rectangular Masks of all targets in the region; assuming that there is a point A(x i ,y i ), B(x j ,y j ), the formula of horizontal reasoning is:

[0016]

[0017] The formula for the vertical reasoning is:

[0018]

[0019] Among them, S h Indicates the horizontal spacing of array targets, θ h Indicates the horizontal azimuth, S v Indicates the vertical distance of the target, θ v Indicates the vertical azimuth;

[0020] First, perform horizontal reasoning to obtain a row of target centroid points, and then perform vertical reasoning based on this to obtain a set of four regional target centroid points M. i (i=1,…,4);

[0021] Concatenate the point sets by rows and columns, and remove the overlapping points to obtain an ordered two-dimensional point set M;

[0022] After obtaining the point set M, a rectangular Mask is generated based on the centroid to segment the target. Its width is consistent with the horizontal spacing of the target, and its height is consistent with the vertical spacing of the target.

[0023] For each region R in the array i , and finally achieve the target global segmentation.

[0024] Optionally, the boundary-aware SegFormer network includes: a prediction module and an improved gated residual refinement module;

[0025] The prediction module is built using SegFormer's MiT-B0 lightweight encoder and MLP decoder;

[0026] The improved gated residual refinement module RRM_GN is based on the prediction result S of the prediction module. coarse As input, learn S based on the U-shaped architecture coarse The residual S between the real label GT residual , thereby refining the results of the prediction module;

[0027] The RRM_GN module is in S coarse Before the input and output of S, a 3×3 convolution kernel is used to adjust the number of channels; coarse With S residual After adding, use the Softmax function to replace the Sigmoid function;

[0028] When downsampling, 3×3 convolution, batch normalization, linear rectification function ReLu and non-overlapping maximum pooling are used;

[0029] After upsampling, recursive gated convolution gnConv of order 5 to 2 is selected from bottom to top to realize spatial interaction of different orders. The normalization method and activation function are the same as the upsampling process, and the upsampling method is bilinear upsampling.

[0030] Optionally, a hybrid loss function is used to supervise the output of the refinement module during the training process of the boundary-aware SegFormer network, and the expression of the hybrid loss function is:

[0031] L hybrid =Al ce +Bl iou +Cl ab

[0032] Among them, l ce is the cross entropy loss, l iou is the intersection-over-union loss, l ab is the active boundary loss, and A, B, C are the weights of the corresponding losses.

[0033] Optionally, the cross entropy loss is:

[0034]

[0035] Among them, N represents the number of samples, C is the number of categories, and p ic Represents the probability of predicting the i-th sample corresponding to category c, y ic ∈{0,1} represents the one-hot encoding of the true label transformation.

[0036] Optionally, the intersection-over-union loss is:

[0037]

[0038] Where C is the number of categories, m(c) is the prediction error for category c, is the Jaccard loss The lovasz extension.

[0039] Optionally, the active boundary loss is:

[0040]

[0041] Among them, N b is the number of pixels on the predicted boundary, Λ(Mi) is the weight of the loss corresponding to pixel i, and M i is the distance from pixel i to the actual boundary; if M i If it is 0, it means that the pixel is already on the real boundary, and ABL will no longer calculate the loss of this pixel. Let the pixel j be the closest to the real boundary in the 8-domain of pixel i on the predicted boundary. Represents the 8-dimensional vector consisting of the KL divergence of pixel i and its 8 neighboring pixels on the predicted boundary; Represents the 8-dimensional vector consisting of the offset direction of pixel i to j, and is the target distribution in the weighted cross entropy loss.

[0042] A second object of the present invention is to provide a wafer array target image segmentation system, which is used to implement any of the above array target image segmentation methods, comprising:

[0043] An image acquisition module is used to collect array target images and perform preprocessing;

[0044] Adaptive seed search module, which searches for ideal seed targets in the preprocessed image area;

[0045] The global segmentation module takes each ideal seed target as the starting point, predicts the centroid position and rectangular Mask of all targets, and realizes the global segmentation of targets in the region Ri;

[0046] The local segmentation module uses a boundary-aware SegFormer network to accurately segment the global segmentation result to obtain the segmentation result of the array target image.

[0047] The third object of the present invention is to provide a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the array target image segmentation method according to any one of the above items is implemented.

[0048] The beneficial effects of the present invention are:

[0049] Aiming at the problem of low segmentation accuracy caused by uneven background, defect interference and weak edges in wafer array target images, this paper proposes an array target image segmentation method based on boundary-aware SegFormer network. The designed global segmentation method makes full use of the distribution characteristics of the target, adaptively searches for multiple ideal seed targets and then performs parallel reasoning, which reduces the reasoning error and achieves efficient global segmentation. In the local segmentation process, the BA-SegFormer semantic segmentation network proposed in this paper achieves high-precision boundary segmentation.

[0050] In one embodiment, an improved gated residual refinement RRM_GN module is used in the BA-SegFormer semantic segmentation network to enhance the target edge features with the help of high-order spatial interactions, and the network is guided by a hybrid loss function to further improve the accuracy of edge segmentation.

[0051] Comparative experiments show that the adaptive seed search strategy proposed in the present invention has a higher success rate and the shortest time compared with the existing methods. The proposed BA-SegFormer network has the best segmentation accuracy and boundary quality compared with the existing segmentation methods, while also maintaining a better network complexity and segmentation speed. The segmentation method of the present invention can completely and accurately segment targets in high-resolution array target images with uneven backgrounds, defect pollution, and low edge contrast, and has high real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 It is a schematic diagram of boundary positioning and partitioning.

[0054] Figure 2 The following are examples of abnormalities on the grain surface, where (a) is the LG wafer area array image; (b) is the LG wafer local line scan image; (c) is the background grayscale unevenness; (d) is the ink dot contamination; (e) is the noisy edge; and (f) is the weak edge.

[0055] Figure 3 It is the flow chart of array target image segmentation of the present invention.

[0056] Figure 4 It is a flow chart of the adaptive seed search of the present invention.

[0057] Figure 5 Schematic diagram of two-point horizontal and vertical reasoning of the present invention, wherein (a) is horizontal reasoning; (b) is vertical reasoning.

[0058] Figure 6 This is the BA-SegFormer network structure diagram of the present invention.

[0059] Figure 7 It is a structural diagram of RRM_GN of the present invention.

[0060] Figure 8 This is a comparison chart of the present invention and other different methods on the Cityscapes dataset.

[0061] Fig. 9 This is a comparison chart of the present invention and other different methods on a grain data set. DETAILED DESCRIPTION

[0062] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0063] Embodiment 1:

[0064] This embodiment provides a wafer array target segmentation method based on a boundary-aware SegFormer network, see Figure 1 , the method comprising:

[0065] Step 1: Collect the wafer array target image, divide the array target image into m regions, and i Parallel processing, i = 0, 1, ..., m;

[0066] Step 2: Adopting an adaptive seed search strategy to search for an ideal seed target in the region, the adaptive seed search strategy comprising: constructing a seed distribution heat map using the correlation between the seed position and the target positioning accuracy, and adaptively searching for the ideal seed target under the guidance of the heat map;

[0067] Step 3: Taking each ideal seed target as the starting point, predict the centroid position and rectangular Mask of all targets to achieve region R i Global segmentation of internal objects;

[0068] Step 4: Use the boundary-aware SegFormer network to accurately segment the global segmentation result and obtain the segmentation result of the wafer array target image.

[0069] Embodiment 2:

[0070] This embodiment provides a wafer array target segmentation method based on a boundary-aware SegFormer network, see Figure 1 , the method comprising:

[0071] Step 1: For the array target, predetermine the number of divided areas and i (i=0,1,…,m, m is the number of regions), high-resolution images are collected and processed in parallel;

[0072] Step 2: Determine the ideal seed target with less interference from complex background and defects based on the adaptive seed search strategy to reduce the reasoning error;

[0073] Step 21: Locate area R first i At the image boundary, four strip-shaped regions of interest (ROIs) are taken that are parallel to the boundary. Figure 1 (a)), then multi-target template matching is used to obtain the target centroid sets (target center point sets) in the four ROIs, and the centroids close to the image boundary are taken to form the array area R i The rectangular boundary of the ; finally, it is divided into 4 sub-rectangular areas ( Figure 1 (b)), there is overlap between regions to avoid losing targets at region boundaries;

[0074] Step 22: Generate a seed distribution heat map using the correlation between seed position and target positioning accuracy, select the seed with the highest weight as the initial seed, and output its actual position in the image; the positioning accuracy is obtained based on the relative position of the target in the array, and the weight is obtained based on the positioning accuracy. The higher the accuracy, the higher the weight.

[0075] Step 23: Generate a ROI twice the size of the template based on the actual position of the initial seed in the image, and then locate the target based on template matching. If the normalized correlation coefficient is less than the set threshold, search for the next candidate seed target based on the heat map until four ideal seed targets are successfully located. Step 3: Take each ideal seed target as the starting point, quickly predict the centroid position and rectangular Mask of all targets, and achieve region R i Global segmentation of internal objects.

[0076] "Normalized mutual correlation coefficient" is a parameter for template matching. When it is greater than the set threshold, the positioning is judged to be successful. When it is less than the threshold, the positioning fails. At this time, it is necessary to select the seed target with the second highest weight based on the seed distribution heat map, and then perform template matching. By analogy, the four seed target positions are obtained.

[0077] The distance and azimuth between the ideal seed and the adjacent points are calculated respectively. Taking each ideal seed target as the starting point, reasoning along the horizontal and vertical directions is performed to quickly predict the centroid position and rectangular Mask of all targets in the area, so as to achieve the area R i Global segmentation of internal objects, such as Figure 5 The inference formula is as follows:

[0078] Horizontal Reasoning:

[0079]

[0080] Vertical Reasoning:

[0081]

[0082] The reasoning starts from the seed point. First, we use equation (1) to reason horizontally to get a row of target centroid points. Based on this, we use equation (2) to reason vertically to get four regional target centroid point sets Mi (i = 1, ..., 4).

[0083] Next, the point set is concatenated by row and column, and the overlapping points are removed to obtain an ordered two-dimensional point set M. After obtaining the point set M, a rectangular Mask is generated based on the centroid to segment the target. Its width is consistent with the horizontal spacing of the target, and its height is consistent with the vertical spacing of the target. i , and finally achieve the target global segmentation.

[0084] Step 4: Design the BA-SegFormer network to perform local precise segmentation on the global segmentation results, remove redundant background and improve boundary segmentation accuracy.

[0085] like Figure 6 As shown in the figure, the model consists of prediction and refinement modules. The prediction module is built using SegFormer's MiT-B0 lightweight encoder and MLP decoder. MiT-B0 is a hierarchical encoder based on Transformer that can output multi-scale features and does not require position encoding, with powerful feature extraction capabilities. The MLP decoder uses 1×1 convolution to unify the channel dimensions of the multi-level features output by the encoder, and then splices the feature layer after upsampling it to 1 / 4 the size of the original image. After feature fusion at the linear layer, a rough prediction result (Coarse Map) is output.

[0086] In order to improve the segmentation accuracy of the target boundary, an improved gated residual refinement module (RRM_GN) is introduced. RRM_GN takes Coarse Map as input and learns the residual between Coarse Map and the true label (GT) based on the U-type architecture to refine the result of the prediction module.

[0087] The improved RRM_GN module fully utilizes the locality of 3×3 convolution during downsampling, and uses gnConv of different orders to achieve high-order spatial interaction after upsampling. After fusing features of different levels, it better learns the residual between Coarse Map and GT and improves the edge segmentation accuracy.

[0088] Figure 7 The improved RRM_GN structure is shown. Before the input and output of the Coarse Map, a 3×3 convolution kernel is used to adjust the number of channels. In order to adapt to multi-label classification, S coarse With S residual After adding, the Softmax function is used to replace the Sigmoid function. When downsampling, 3×3 convolution, batch normalization, linear rectification function (ReLu) and non-overlapping maximum pooling are still used, where S coarse Refers to the above Coarse Map, which is used as the input of the RRM_GN module. residual It is the partial output of RRM_GN, that is, the residual between the above Coarse Map and the true label GT.

[0089] After upsampling, recursive gated convolution gnConv of order 5 to 2 is selected from bottom to top to achieve spatial interaction of different orders. The normalization method and activation function are the same as upsampling. The upsampling method is bilinear upsampling. The output of the RRM_GN module is directly used to segment the target.

[0090] In addition, the output of the refinement module (Refine Map) is supervised during training based on a hybrid loss function. In order to obtain high-precision region segmentation results and clear boundaries, different loss functions are selected for all pixels, the interior of the region, and the region boundary according to the characteristics of the loss function. The active boundary loss (ABL) is introduced to supervise the pixels in the boundary area, and cooperates with the cross entropy loss and the intersection-over-union loss to promote the model to actively predict the boundary and approach the real boundary during the training process. The expression of the hybrid loss function is as follows:

[0091] L hybrid =Al ce +Bl iou +Cl ab , (3)

[0092] Among them, l ce is the cross entropy loss (CE Loss), l iou is the intersection-over-union loss (Lovasz-Softmax Loss), l ab is the Active Boundary Loss (ABL), and A, B, and C are the weights of the corresponding losses.

[0093] CE Loss is the most commonly used loss function for semantic segmentation. It focuses on the classification of all pixels and is defined as:

[0094]

[0095] Among them, N represents the number of samples, C is the number of categories, and p ic Represents the probability of predicting the i-th sample corresponding to category c. ic ∈{0,1} represents the one-hot encoding of the true label transformation.

[0096] Lovasz-Softmax Loss focuses on region-level segmentation and is defined as:

[0097]

[0098] Where C is the number of categories, m(c) is the prediction error for category c, is the Jaccard loss The lovasz extension.

[0099] Active Boundary Loss is defined as:

[0100]

[0101] Among them, N b is the number of pixels on the predicted boundary, Λ(Mi) is the weight of the loss corresponding to pixel i, and M i is the distance from pixel i to the actual boundary. i If it is 0, it means that the pixel is already on the real boundary, and ABL no longer calculates the loss of this pixel. Let the pixel j be the closest to the real boundary in the 8-dimensional region of pixel i on the predicted boundary. Represents the 8-dimensional vector consisting of the KL divergence between pixel i and its 8 neighboring pixels on the predicted boundary. Represents the 8-dimensional vector consisting of the offset direction of pixel i to j, and is the target distribution in the weighted cross entropy loss.

[0102] To verify the effectiveness of the segmentation method proposed in this paper, a grain array dataset (LG type and JD type) was produced based on wafer images. The size of each regional sub-image of the two types of wafers is 8320×45999 and 8230×31800, respectively, and the number of grains contained is 6530 and 8650, respectively. To verify the effectiveness of the BA-SegFormer model, the test results of multiple methods are compared and studied on the self-built grain segmentation dataset, and the analysis is also carried out on the public dataset Cityscapes.

[0103] The experimental environment is: the experimental platform is Windows 10 operating system, Intel Core i9-12900k CPU, GeForce RTX 3090 GPU. The development environment is Python 3.8, Pytorch 1.10.0, CUDA 11.3.

[0104] Comparing the results of different global segmentation methods, the experimental results are shown in Table 1 and Table 2:

[0105] Table 1 Results of different splitting methods for LG type wafers

[0106]

[0107] Table 2 Results of different segmentation methods for JD type wafers

[0108]

[0109] As can be seen from Tables 1 and 2, the adaptive seed search segmentation method of the present invention has the highest success rate and the shortest time, and is suitable for real-time and accurate segmentation of high-resolution array target images. In terms of time, compared with LINE-2D and grayscale template matching methods, the adaptive seed search segmentation method takes only 1 / 19 and 1 / 7 of their time in LG wafers, and 1 / 29 and 1 / 7 of their time in JD type wafers, respectively. Among them, although LINE-2D has a higher success rate, it takes a long time to extract edge shape features and cannot be used for real-time segmentation of high-resolution images. In terms of success rate, Otsu's method and morphological method are greatly affected by surface defects and background unevenness, and have the lowest success rate. The adaptive seed search segmentation method relies on the characteristics of the grain array, and its dependence on image quality is lower than other methods including LINE-2D, so it has the highest success rate.

[0110] In order to verify the effectiveness of the BA-SegFormer method of the present invention, comparative experiments were conducted on the method of the present invention with FCN, PSPNet, Deeplabv3+, and SegFormer methods on the self-built grain dataset and Cityscapes dataset. In order to fairly compare the real-time segmentation performance of the network, the backbone networks of the first three networks were replaced with MobileNetV2 to reduce the complexity of the network and improve the segmentation speed.

[0111] The comparative experimental results on Cityscapes are shown in Table 3, where the parameter quantity (Params) measures the complexity of the network structure and FPS evaluates the segmentation speed.

[0112] Table 3 Comparison results of different methods on Cityscapes dataset

[0113]

[0114] The results show that the proposed method has the highest segmentation accuracy and boundary quality, while the number of parameters and segmentation speed are second only to the original SegFormer, verifying the effectiveness of improving the RRM-GN module and introducing the hybrid loss function.

[0115] The visualization of the results of the segmentation method with higher accuracy, PSPNet, DeepLabV3+, SegFormer and the present invention is shown in the figure. Figure 8 The segmentation method of the present invention has higher boundary quality when segmenting slender objects (such as traffic light poles, guardrails, etc.), and also shows higher accuracy in segmenting small objects (such as distant pedestrians, traffic lights, sidewalks, etc.).

[0116] Fig. 9 The qualitative comparison results of various methods on the grain dataset. In order to more clearly reflect the boundary area segmentation results, some images are cut out. Fig. 9 The rectangular box area in the image is drawn, and a straight line is drawn to represent the real boundary. For grains contaminated by ink dots, FCN and PSPNet cannot resist the influence of defects on the target boundary well, and the boundary segmentation is not fine enough. For grains with shadow interference and low edge contrast, the boundary positions predicted by FCN and PSPNet are quite different from the real boundaries. For different types of defective grains, the method of the present invention relies on the RRM_GN module to better learn the boundary information, and is supplemented by the supervision of a mixed loss function including boundary loss. The segmented target boundary is more continuous and smooth than DeepLabv3+ and SegFormer.

[0117] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0118] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A wafer array target image segmentation method, characterized in that: The method comprises: Step 1: Acquire a wafer array target image, divide the wafer array target image into m regions, and perform a i Parallel processing, i = 0, 1, ..., m; Step 2: Adopting an adaptive seed search strategy to search for an ideal seed target in the region, the adaptive seed search strategy comprising: constructing a seed distribution heat map using the correlation between the seed position and the target positioning accuracy, and adaptively searching for the ideal seed target under the guidance of the heat map; Step 3: Taking each ideal seed target as the starting point, predict the centroid position and rectangular Mask of all targets to achieve region R i Global segmentation of internal objects; Step 4: Use the boundary-aware SegFormer network to accurately segment the global segmentation result to obtain the segmentation result of the wafer array target image; The adaptive seed search strategy includes: Step 21: Locate area R first i The rectangular boundary of the image is obtained, and four strip regions of interest ROIs parallel to the boundary are taken at the image boundary. Then, the target centroid sets in the four ROIs are obtained by multi-target template matching, and the centroids close to the image boundary are taken to form the array area R i Finally, it is divided into 4 sub-rectangular regions, with overlaps between regions to avoid losing the target at the region boundary; Step 22: Generate a seed distribution heat map using the correlation between the seed position and the target positioning accuracy, select the seed with the highest weight as the initial seed, and output its actual position in the image; Step 23: Generate a ROI twice the size of the template according to the actual position of the initial seed in the image, and then locate the target based on template matching. If the normalized cross-correlation coefficient is less than the set threshold, search for the next candidate seed target according to the heat map until four ideal seed targets are successfully located. The step 3 includes: starting from each ideal seed target, reasoning along the horizontal and vertical directions, predicting the centroid positions and rectangular Masks of all targets in the region; assuming that there is a point A(x i ,y i ), B(x j ,y j ), the formula for horizontal reasoning is: The formula for vertical reasoning is: Among them, S h Indicates the horizontal spacing of array targets, θ h Indicates the horizontal azimuth, S v Indicates the vertical distance of the target, θ v Indicates the vertical azimuth; First, perform horizontal reasoning to obtain a row of target centroid points, and then perform vertical reasoning based on this to obtain a set of four regional target centroid points M. i (i=1,…,4); Concatenate the point sets by rows and columns, and remove the overlapping points to obtain an ordered two-dimensional point set M; After obtaining the point set M, a rectangular Mask is generated based on the centroid to segment the target. Its width is consistent with the horizontal spacing of the target, and its height is consistent with the vertical spacing of the target. For each region R in the array i , and finally achieve the target global segmentation; The boundary-aware SegFormer network includes: a prediction module and an improved gated residual refinement module; The prediction module is built using SegFormer's MiT-B0 lightweight encoder and MLP decoder; The improved gated residual refinement module RRM_GN is based on the prediction result S of the prediction module. coarse As input, learn S based on the U-shaped architecture coarse The residual S between the real label GT residual , thereby refining the results of the prediction module; The RRM_GN module is in S coarse Before the input and output of S, a 3×3 convolution kernel is used to adjust the number of channels; coarse With S residual After adding, use the Softmax function to replace the Sigmoid function; When downsampling, 3×3 convolution, batch normalization, linear rectification function ReLu and non-overlapping maximum pooling are used; After upsampling, recursive gated convolution gnConv of order 5 to 2 is selected from bottom to top to achieve spatial interaction of different orders. The normalization method and activation function are the same as the upsampling process, and the upsampling method is bilinear upsampling. The output of the refinement module is supervised by a hybrid loss function during the training process of the boundary-aware SegFormer network. The expression of the hybrid loss function is: <h2 style=";text-align:left;direction:ltr">L<h2 style=";text-align:left;direction:ltr"> hybrid <h2 style=";text-align:left;direction:ltr"> (Al)<h2 style=";text-align:left;direction:ltr"> ce <h2 style=";text-align:left;direction:ltr"> +Bl<h2 style=";text-align:left;direction:ltr"> iou <h2 style=";text-align:left;direction:ltr"> +Cl<h2 style=";text-align:left;direction:ltr"> ab Among them, l ce is the cross entropy loss, l iou is the intersection-over-union loss, l ab is the active boundary loss, A, B, C are the weights of the corresponding losses; The cross entropy loss is: Among them, N represents the number of samples, C is the number of categories, and p ic Represents the probability of predicting the i-th sample corresponding to category c, y ic ∈{0,1} represents the one-hot encoding of the true label conversion; The intersection-over-intersection loss is: Where C is the number of categories, m(c) is the prediction error for category c, is the Jaccard loss lovasz extension; The active boundary loss is: Among them, N b is the number of pixels on the prediction boundary, Λ(M i ) is the weight of the loss corresponding to pixel i, M i is the distance from pixel i to the actual boundary; if M i If it is 0, it means that the pixel is already on the real boundary, and ABL will no longer calculate the loss of this pixel. Let the pixel j be the closest to the real boundary in the 8-domain of pixel i on the predicted boundary. Represents the 8-dimensional vector consisting of the KL divergence of pixel i and its 8 neighboring pixels on the predicted boundary; Represents the 8-dimensional vector consisting of the offset direction of pixel i to j, and is the target distribution in the weighted cross entropy loss.

2. A wafer array target image segmentation system, characterized in that: The system is used to implement the array target image segmentation method as claimed in claim 1, comprising: An image acquisition module is used to collect array target images and perform preprocessing; Adaptive seed search module, which searches for ideal seed targets in the preprocessed image area; The global segmentation module takes each ideal seed target as the starting point, predicts the centroid position and rectangular Mask of all targets, and realizes the global segmentation of targets in the region Ri; The local segmentation module uses a boundary-aware SegFormer network to accurately segment the global segmentation result to obtain the segmentation result of the array target image.

3. A computer-readable storage medium storing computer-executable instructions, wherein when the computer-executable instructions are executed by a processor, the wafer target image segmentation method according to claim 1 is implemented.

Images