A multi-scale chip defect size measurement method based on UNet++

Abstract

This invention discloses a multi-scale chip defect size measurement method based on UNet++. The method constructs a multi-category, multi-scale semantic segmentation dataset for chip surface defect images and trains a UNet++ semantic segmentation network to obtain a segmentation model. The chip surface defect images are input into the segmentation model to obtain feature maps of different scales. These maps are then input into a prediction head based on an attention mechanism. The connected domains of defects of each category in the segmented image are extracted, and the defect outlines and skeletons are annotated. The multi-scale defect feature-annotated images are then input into a multi-scale attention feature fusion module. The contour point sets of depressions and stains are extracted from the fused image, and the pixel-level perimeter, area, and pixel-level length are calculated. The image defect size information is then converted into the actual defect size information based on the ratio of pixel length to actual length. By introducing the attention mechanism and multi-scale feature fusion, the present invention effectively improves the segmentation and measurement accuracy of multi-scale chip defects.

Description

Technical Field

[0001] The present invention belongs to the field of image or video recognition or understanding technology, and specifically relates to a multi-scale chip defect size measurement method based on UNet++. Background Art

[0002] With the rapid development of the chip industry, the importance of chip surface defect detection and dimensional measurement in chip manufacturing has become increasingly prominent. Traditional chip surface defect dimensional measurement methods rely primarily on manual feature extraction and rule definition. For example, these methods involve manually observing chip surface defects, determining the defect type based on experience, and performing rough dimensional measurements. This approach has numerous drawbacks. Firstly, it relies heavily on specialized knowledge and requires a high level of operator experience and skill. Secondly, it is extremely difficult to accurately identify complex defects, making them prone to misjudgments and omissions.

[0003] Current approaches to measuring chip surface defect sizes primarily include manual measurement and the use of traditional image processing, machine learning, and deep learning methods to measure defect images. Manual measurement is not only inefficient but also highly subjective, with different measurement personnel likely obtaining different results. Traditional image processing methods, such as edge detection and threshold segmentation, often suffer from low accuracy when dealing with complex chip surface defects, making it difficult to accurately segment defect areas. While machine learning methods have improved defect detection accuracy to a certain extent, their adaptability to defects of varying sizes and shapes is limited. For example, support vector machines (SVMs) are effective for defects of certain specific shapes, but their generalization capabilities are insufficient for complex and variable chip surface defects. Traditional neural network models, on the other hand, are unstable when dealing with defects of varying sizes and shapes, failing to achieve high-precision segmentation and size measurement.

[0004] In summary, existing technical methods have problems such as low accuracy, strong subjectivity, and insufficient generalization ability in chip surface defect measurement, making it difficult to meet the growing demand for chip surface defect measurement. Summary of the Invention

[0005] In order to solve the problems existing in the prior art, the purpose of the present invention is to provide a multi-scale chip defect size measurement method based on UNet++. By constructing a multi-category, multi-scale semantic segmentation dataset of chip surface defect images to train the UNet++ semantic segmentation network, a prediction head based on the attention mechanism is designed to extract multi-scale feature maps from the decoder part of the semantic segmentation model and generate multi-scale semantic segmentation images. A multi-scale attention feature fusion module is designed to fuse multi-scale semantic segmentation images. These two modules enhance the defect extraction accuracy and multi-scale generalization ability of the present method, and solve the problems of low accuracy, strong subjectivity and lack of generalization ability of existing measurement technologies in chip surface defect measurement.

[0006] The present invention is achieved through the following technical solutions.

[0007] One aspect of the present invention provides a multi-scale chip defect size measurement method based on UNet++, comprising the following steps:

[0008] Construct a multi-category, multi-scale semantic segmentation dataset of chip surface defect images, use the dataset to train the UNet++ semantic segmentation network, and obtain the UNet++ semantic segmentation model;

[0009] The chip surface defect image is input into the UNet++ semantic segmentation model. Feature maps of different scales are obtained from the decoder part of the segmentation model. The feature maps of different scales are input into the prediction head based on the attention mechanism to obtain multi-scale semantic segmentation images.

[0010] Extract the connected domains of each defect category in the multi-scale semantic segmentation image. For the connected domains of depressions and stains, use the contour approximation algorithm to annotate the defect contours. For the connected domains of abrasions and scratches, use the skeleton extraction algorithm to annotate the defect skeletons, thus obtaining a multi-scale defect feature annotated image.

[0011] Input the multi-scale defect feature annotated image into the multi-scale attention feature fusion module to obtain the fused defect feature annotated image;

[0012] Extract the contour point set of dents and stains from the fused defect feature annotation image, calculate the perimeter and area of ​​the image pixel level, and extract the skeleton point set of abrasions and scratches to calculate the image pixel level length;

[0013] According to the ratio of the chip surface defect image pixel length to the actual length, the defect size information at the image pixel level is converted into the actual size information of the image defect.

[0014] Preferably, the constructing of a multi-category, multi-scale semantic segmentation dataset of chip surface defect images includes:

[0015] Use optical cameras and optical microscopes to collect images of chip surface defects of different sizes and categories;

[0016] Preprocessing the collected chip surface defect images, including bilateral filtering and histogram equalization;

[0017] Performing dataset expansion on the preprocessed image to obtain an expanded dataset;

[0018] The images in the amplified dataset are annotated with pixel-level semantic segmentation and divided into training and test sets to obtain a semantic segmentation dataset of chip surface defect images.

[0019] Preferably, feature maps of different scales are obtained from the decoder part of the segmentation model, including:

[0020] For the input chip surface defect image, the encoder structure of the UNet++ semantic segmentation model is used to downsample the defect image, extract the features of the defect image, and obtain the encoder feature output;

[0021] For the encoder feature output, the decoder structure and dense skip connection structure of the UNet++ semantic segmentation model are used to perform feature upsampling and feature fusion to obtain the decoder feature output;

[0022] Get feature maps of different scales from the decoder part of the segmentation model.

[0023] As a preference, feature maps of different scales are obtained from the decoder part of the segmentation model, specifically the feature maps of the last three layers of the decoder, whose sizes are H×W×64, and The sizes of the semantic segmentation images after the prediction head based on the attention mechanism are: H×W×3, and Where H and W are the vertical and horizontal pixel numbers of the input image, respectively.

[0024] Preferably, the structure of the prediction head based on the attention mechanism includes: a scale-aware attention module SAAM for extracting multi-scale effective features; a 1×1 convolution for adjusting the number of channels; and a Softmax classifier for classifying pixels.

[0025] The structure of the scale-aware attention module SAAM includes: a channel attention module CAB, which is used to adjust the degree of attention to each channel; a spatial attention module SAB, which is used to enhance the module's ability to pay attention to specific locations; and a scale convolution module SCM, which is used to enhance the module's representation of features and capture multi-scale and multi-resolution backgrounds.

[0026] Preferably, feature maps of different scales are input into the prediction head based on the attention mechanism to obtain multi-scale semantic segmentation images, including:

[0027] The feature maps are input into the scale-aware attention module SAAM respectively, and the channel attention module CAB and the spatial attention module SAB are used to enhance the attention to specific channels and spaces, and the scale convolution module SCM is used to enhance the multi-scale representation of features;

[0028] Input the feature map into 1×1 convolution for feature extraction and adjust the number of channels;

[0029] The feature map is input into the softmax classifier to classify the pixels and obtain semantic segmentation images of different scales.

[0030] Preferably, obtaining a multi-scale defect feature annotated image includes:

[0031] Grayscale the semantic segmentation images of different scales to extract the connected domains of various defects;

[0032] Create blank images of corresponding sizes for semantic segmentation images of different scales respectively;

[0033] For the connected areas of depressions and stains, the Douglas-Peucker algorithm is used to perform contour approximation processing to obtain the contour point sets of depressions and stains, and mark them on the corresponding blank images;

[0034] For the connected domains of abrasions and scratches, the Zhang-Suen thinning algorithm is used to extract the skeleton to obtain the skeleton point sets of abrasions and scratches, and annotate them on the corresponding blank images.

[0035] Preferably, the structure of the multi-scale attention feature fusion module includes:

[0036] Two upsampling modules to resize feature maps to the same size;

[0037] A multi-feature fusion module MFFM, used to fuse multiple feature maps;

[0038] A combination module of two 3×3 convolutions, Batch Normalization, and ReLU activation functions for further feature extraction and channel adjustment;

[0039] The structure of the multi-feature fusion module MFFM includes:

[0040] A multi-feature attention gating module (MFAG) that enhances low-level semantic features of the original feature map by upsampling features and uses grouped convolution to reduce computational complexity;

[0041] A channel attention module CAB and a spatial attention module SAB are used to highlight the important channels and spatial positions of the target area in the feature map and improve the ability to express features.

[0042] Preferably, the multi-scale feature annotated image is input into a multi-scale attention feature fusion module, including:

[0043] The feature map is resized through the upsampling module;

[0044] The feature map is input into the multi-feature fusion module MFFM for feature fusion to obtain the feature map;

[0045] After two 3×3 convolutions, Batch Normalization and ReLU activation function combination modules, further feature extraction and channel adjustment are performed to obtain the defect feature annotation image.

[0046] The present invention adopts the above technical solution, which has the following beneficial effects:

[0047] 1) To address the problem of weak generalization ability of traditional segmentation methods, the present invention trains a UNet++ deep learning semantic segmentation model based on a chip surface defect semantic segmentation dataset, and designs a prediction head based on an attention mechanism to obtain multi-scale feature maps from the model's decoder to generate multi-scale semantic segmentation images. The scale-aware attention module SAAM in the prediction head enhances the ability to focus on key channels and positions in the features through channel and spatial attention, thereby enhancing the representation of features; the scale convolution module SCM maps features to different scales through multi-scale deep convolution, thereby capturing and combining multi-resolution information. The prediction head based on the attention mechanism enhances the understanding of important contextual information, and the generated multi-scale semantic segmentation image is used to improve the generalization ability of defect segmentation.

[0048] 2) To address the low precision of traditional defect feature extraction methods, the present invention generates multi-scale defect feature annotated images and designs a multi-scale attention feature fusion module to perform feature fusion on the annotated images. The multi-feature fusion module MFFM in the multi-scale attention feature fusion module combines multiple feature maps through the multi-feature attention gating module MFAG, uses upsampled high-level semantic features to enhance the low-level semantic features of the original-size image, fuses information from different scales, and reduces information loss; then, through channel and spatial attention, adjusts the ability to focus on important channels and spatial positions in the target area, improving the present invention's ability to express defect features and the accuracy of feature extraction.

[0049] 3) In order to solve the problem that the existing defect size measurement methods have a single and non-targeted measurement method for different defects, the present invention designs different defect measurement methods for different types of defects, thereby improving the targeting and applicability of defect size measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute an improper limitation of the present invention. In the drawings:

[0051] Figure 1 Schematic diagram of the process of a multi-scale chip defect size measurement method based on UNet++ in a preferred embodiment of the present invention;

[0052] Figure 2 Schematic diagram of the structure of a prediction head based on an attention mechanism in a preferred embodiment of the present invention;

[0053] Figure 3 Schematic diagram of the structure of the scale-aware attention module SAAM in a preferred embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of the structure of a multi-scale attention feature fusion module in a preferred embodiment of the present invention;

[0055] Figure 5 Schematic diagram of the structure of a multi-feature fusion module MFFM in a preferred embodiment of the present invention;

[0056] Figure 6 Schematic diagram of the overall structure of a multi-scale chip defect size measurement method based on UNet++ in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0057] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The exemplary embodiments and descriptions of the present invention are used to explain the present invention but are not intended to limit the present invention.

[0058] like Figure 1 As shown, the multi-scale chip defect size measurement method based on UNet++ provided by the present invention includes the following steps:

[0059] S101: Build a multi-category, multi-scale semantic segmentation dataset of chip surface defect images, use the dataset to train the UNet++ semantic segmentation network, and obtain the UNet++ semantic segmentation model.

[0060] The specific steps include:

[0061] S11, uses an optical camera and an optical microscope to collect images of chip surface defects of different sizes and categories;

[0062] S12, preprocessing the collected chip surface defect image, using the bilateralFilter() function in the OpenCV library to perform bilateral filtering to retain the image edge information; then using the equalizeHist() function in the OpenCV library to perform histogram equalization to enhance the image contrast;

[0063] S13, expanding the dataset of the preprocessed image, using functions such as blur(), flip(), and warpAffine() in the OpenCV library to perform operations such as blurring, mirroring, and rotating the image to obtain an expanded dataset;

[0064] S14, use automated annotation software to perform pixel-level semantic segmentation annotation on the images in the amplified dataset, annotate the segmentation mask and category label for each defect category, and then convert the annotated json file into a PNG image file that can be used for semantic segmentation training. Then divide the training set and test set to obtain the semantic segmentation dataset of chip surface defect images.

[0065] S15, using the semantic segmentation training set of chip surface defect images, uses the resize() function in the OpenCV library to resize the input image to 512×512 as the input for model training;

[0066] In step S16, VGG16 and ResNet50 are selected as the backbone networks of UNet++, and the training weights of the backbone networks on the Imagenet dataset are loaded as pre-trained weights. SGD is selected as the optimizer, and appropriate training parameters such as learning rate and loss function are selected to train the UNet++ semantic segmentation network.

[0067] S17, using the chip surface defect image semantic segmentation test set to evaluate the performance of the trained UNet++ semantic segmentation model, and obtain the model's mIoU, mPA and other indicators;

[0068] In S18, adjust the training parameters such as the backbone network, learning rate, and loss function, repeat the training and evaluation process, and select the UNet++ semantic segmentation model with the best performance.

[0069] In the above steps, a multi-category, multi-scale semantic segmentation dataset of chip surface defect images was constructed. Through optical acquisition, preprocessing, and data amplification, image quality and diversity were ensured, thereby enhancing the model's training effectiveness. To address the weak generalization and low segmentation accuracy of traditional methods, a UNet++ deep learning semantic segmentation model was trained on this dataset. By selecting appropriate backbone network parameters and performing multiple adjustments and evaluations, the model's segmentation accuracy for defects of different types and scales was improved, enhancing the generalization and robustness of the segmentation method.

[0070] S102: Input the chip surface defect image into the UNet++ semantic segmentation model, obtain feature maps of different scales from the decoder part of the segmentation model, and input the feature maps of different scales into the prediction head based on the attention mechanism to obtain a multi-scale semantic segmentation image. The specific steps include:

[0071] S21, for the input chip surface defect image, downsample it using the encoder structure of the UNet++ semantic segmentation model to extract the features of the defect image and obtain the encoder feature output;

[0072] S22, for the encoder feature output, use the decoder structure of the UNet++ semantic segmentation model and the dense skip connection structure to perform feature upsampling and feature fusion to obtain the decoder feature output;

[0073] S23, obtains feature maps of different scales from the decoder part of the segmentation model, specifically the feature maps of the last three layers of the decoder, with sizes of H×W×64, and They are input into the prediction head based on the attention mechanism, and the final sizes are H×W×3, and where H and W are the number of vertical and horizontal pixels of the input image, respectively.

[0074] like Figure 2 As shown in the figure, the structure of the prediction head based on the attention mechanism includes: a scale-aware attention module SAAM for extracting multi-scale effective features; a 1×1 convolution for adjusting the number of channels; and a Softmax classifier for classifying pixels. The expression of the prediction head based on the attention mechanism is as follows:

[0075] Head=Softmax(Conv 1×1 (SAAM(x)))

[0076] Where, Softmax() represents the Softmax classifier; Conv 1×1 () represents 1×1 convolution.

[0077] like Figure 3 As shown in Figure 2, the structure of the scale-aware attention module SAAM includes: a channel attention module CAB, which is used to adjust the attention level of each channel; a spatial attention module SAB, which is used to enhance the module's ability to focus on specific locations; and a scale convolution module SCM, which is used to enhance the module's representation of features and capture multi-scale and multi-resolution context. The expression of SAAM is as follows:

[0078] SAAM=SCM(SAB(CAB(x)))

[0079]

[0080] SCM=BN(C1(CS(∑ ks∈Ks ReLU(BN(DWC ks (ReLU(BN(C3(x)))))))))+x

[0081] Where x represents the feature map; Sigmoid() represents the Sigmoid activation function; C1 represents a 1×1 convolution with the same number of channels as the original number of channels; C2 represents a 1×1 convolution with the same number of channels as 1 / 16 of the original number of channels; C3 represents a 1×1 convolution with the same number of channels as twice the original number of channels; ReLU() represents the ReLU activation function; P m () represents adaptive average pooling; P a () indicates adaptive maximum pooling; Indicates element-by-element multiplication; LKC() indicates 6×6 convolution; Chmax() indicates channel maximum; Chavg() indicates channel average; BN() indicates batch normalization; CS() indicates channel shuffling; DWC ks () represents the depth convolution with convolution kernel ks; KS represents the set of convolution kernels.

[0082] The multi-scale feature maps are input into the prediction head based on the attention mechanism, including:

[0083] First, the size is H×W×64, and The feature maps are input into the scale-aware attention module SAAM respectively, and the ability to focus on specific channels and spaces is enhanced by CAB and SAB in turn, and the multi-scale representation of features is enhanced by SCM; then the feature map is input into 1×1 convolution for feature extraction and the number of channels is adjusted; finally, the feature map is input into the softmax classifier to classify the pixels, and different colors are marked for pixels of different categories, and finally three semantic segmentation images of different scales are obtained, with sizes of H×W×3, and Where H and W are the vertical and horizontal pixel numbers of the input image, respectively.

[0084] By acquiring feature maps from the last three layers of the decoder, we can fully utilize feature information at different levels. These maps are then fed into the Scale-Aware Attention Module (SAAM). Through channel and spatial attention, they enhance focus on key channels and locations within the features, respectively, thereby enhancing feature representation. The Scaled Convolution Module (SCM) maps features to different scales, capturing and combining multi-resolution information. The resulting feature maps are then passed through convolution and a classifier to generate semantically segmented images at three different scales, improving the generalization of multi-scale features.

[0085] S103, extracting the connected domains of each category of defects in the multi-scale semantic segmentation image, using a contour approximation algorithm to annotate the defect contours for the connected domains of dents and stains, and using a skeleton extraction algorithm to annotate the defect skeletons for the connected domains of abrasions and scratches, to obtain a multi-scale defect feature annotated image.

[0086] The specific steps include:

[0087] S31, grayscale processing is performed on the semantic segmentation images of three different scales to extract the connected domains of various defects;

[0088] S32, creates blank images of corresponding sizes for the semantic segmentation images of three different scales;

[0089] S33, for the concave and stain connected domains, use the Douglas-Peucker algorithm to perform contour approximation processing to obtain the contour point set of the object and mark it on the corresponding blank image;

[0090] S34, for the connected areas of abrasions and scratches, use the Zhang-Suen thinning algorithm to extract the skeleton, obtain the skeleton point set of the object, and mark it on the corresponding blank image.

[0091] After grayscale processing of the multi-scale semantic segmentation image, the connected domains of various defects are extracted. Different annotation methods are used for different types of defects, providing accurate multi-scale defect feature annotation images for the subsequent accurate calculation of the defect size.

[0092] S104: Input the multi-scale feature-annotated image into a multi-scale attention feature fusion module to obtain a fused defect feature-annotated image.

[0093] like Figure 4As shown in Figure 1, the structure of the multi-scale attention feature fusion module includes: two upsampling modules for adjusting feature maps to the same size; a multi-feature fusion module MFFM for fusing multiple feature maps; and two 3×3 convolution, batch normalization, and ReLU activation function combination modules for further feature extraction and channel adjustment. The expression of the multi-scale attention feature fusion module is as follows:

[0094] Feature Fusion=G(G(MFFM(x,US(y),US(z))))

[0095] G(x)=ReLU(BN(Conv 3×3 (x)))

[0096] Where x is the feature map of size H×W×3; y is Characteristic map of size; z is The feature map of the size is 100; US() is the upsampling operation; the definitions of ReLU() and BN() are the same as those in the prediction head based on the attention mechanism.

[0097] like Figure 5 As shown in Figure 2, the structure of the multi-feature fusion module MFFM includes: a multi-feature attention gating module MFAG, which enhances the low-level semantic features of the original-size feature map by upsampling features and uses grouped convolution to reduce computational complexity; a channel attention module CAB and a spatial attention module SAB, which are used to highlight the important channels and spatial positions of the target area in the feature map, thereby improving the expressiveness of the features. The expression of the multi-scale attention feature fusion module is as follows:

[0098] MFFM=SAB(CAB(ConCat(MFAG(x,y,z),y,z)))

[0099] MFAG=x×(1+Sigmoid(Conv 1×1 (ReLU(H(x)+H(y)+H(z)))))

[0100] H(x)=ReLU(BN(GroupConv 1×1 (x)))

[0101] In the formula, ConCat() is feature concatenation; GroupConv 1×1 () is a 1×1 group convolution; the definitions of x, y, and z are the same as those in the multi-scale attention feature fusion module; SAB(), CAB(), Sigmoid(), Conv 1×1 The definitions of (), ReLU() and BN() are the same as those in the prediction head based on the attention mechanism.

[0102] The multi-scale feature annotated image is input into the multi-scale attention feature fusion module, which includes:

[0103] First and The feature map of size is adjusted to H×W×3 through the upsampling module; then the three feature maps are input into the multi-feature fusion module MFFM for feature fusion to obtain a feature map of size H×W×6; finally, a combination module of two 3×3 convolutions, Batch Normalization and ReLU activation function is used to further extract features and adjust channels, and finally a defect feature annotation image of size H×W×3 is obtained.

[0104] The multi-scale feature-annotated image is input into the multi-scale attention feature fusion module for feature fusion. The multi-feature fusion module MFFM combines multiple feature maps through the multi-feature attention gating module MFAG, and uses the upsampled high-level semantic features to enhance the low-level semantic features of the original size, fusing information from different scales and reducing information loss. It also adjusts the ability to focus on important channels and spatial positions in the target area through channel and spatial attention, thereby improving the ability to express defect features.

[0105] S105, extracting the contour point set of the dents and stains from the fused defect feature annotated image to calculate the pixel-level perimeter and area, and extracting the skeleton point set of the abrasions and scratches to calculate the pixel-level length, specifically includes the following steps:

[0106] S51, extracting the contour point sets of the depressions and stains from the fused defect feature annotated image, and calculating the sum of the Euclidean distances of adjacent pixels in each defect object contour point set according to the following formula to obtain the perimeter of the defect object;

[0107]

[0108] Where N is the sum of the Euclidean distances between all points n; x i and y i 、x j and y j Represent the x-coordinate and y-coordinate of the i-th and j-th points respectively.

[0109] The double integral of the area enclosed by the contour point set of each defect object is calculated according to the following formula to obtain the area of ​​the defect;

[0110]

[0111] Where S represents the curve integral result; f(x,y) and g(x,y) represent two scalar functions defined on the region; and Represents the partial derivative of the function g(x,y) with respect to x and the partial derivative of the function f(x,y) with respect to y.

[0112] S52. Extract the skeleton point set of the abrasions and scratches from the fused defect feature annotated image, and calculate the sum of the Euclidean distances of adjacent pixels in the skeleton point set of each defect object according to the following formula to obtain the length of the defect object;

[0113]

[0114] Where L represents the sum of the Euclidean distances between all points; x i and y i 、x j and y j Represent the x-coordinate and y-coordinate of the i-th and j-th points respectively.

[0115] Corresponding size calculation methods are provided for different types of defects, enhancing the pertinence of size measurement.

[0116] S106, obtaining the pixel length corresponding to the known length in the chip surface defect image, calculating the ratio of the actual length to the pixel length, and converting the image pixel-level defect size information into the actual size information of the image defect.

[0117] The specific steps include:

[0118] S61, obtain the pixel length corresponding to a known length (such as a ruler) in the image, and calculate the ratio of the actual length to the pixel length, that is, the ratio L of the actual length (mm) corresponding to the length of one pixel. pp ; The calculation formula for size conversion is:

[0119]

[0120] Where, L pp is the physical size of each pixel, L d is the shooting distance, L f is the focal length of the camera, P c It is the number of pixels corresponding to 1 cm of the camera's light-receiving element.

[0121] S62: Multiply the defect perimeter and length of the object calculated above by the scale, and multiply the area of ​​the defect object by the square of the scale to obtain the actual size information of the defect.

[0122] By obtaining the pixel length corresponding to the known length scale in the image, calculating the ratio between the actual length and the pixel length, and converting the calculated pixel-level dimension information into actual length and area, this step ensures the practical application value and accuracy of the measurement results, meeting actual production needs.

[0123] The overall structural diagram of the multi-scale chip defect size measurement method based on UNet++ provided by the present invention is as follows: Figure 6 shown.

[0124] This paper constructs a multi-category, multi-scale semantic segmentation dataset of chip surface defect images, trains a UNet++ semantic segmentation network, and designs an attention-based prediction head and a multi-scale attention feature fusion module to extract and fuse multi-scale feature maps. Furthermore, it designs appropriate measurement algorithms for different defect types, ultimately addressing the low precision and subjectivity of chip surface defect measurement, as well as the lack of generalization capabilities of existing measurement techniques. The proposed method has clear physical significance, high reliability, and robustness, providing an effective technical means for chip surface defect measurement.

[0125] It should be noted that the above embodiment is a preferred implementation and should not be construed as limiting the present invention. Professionals and technicians can further implement the present invention using electronic hardware, computer software, or a combination of both, by combining the modules and algorithm steps of each example described in the embodiments disclosed herein. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0126] The present invention is not limited to the above-mentioned embodiments. On the basis of the technical solutions disclosed in the present invention, those skilled in the art can make some substitutions and modifications to some of the technical features therein according to the disclosed technical content without creative labor, and these substitutions and modifications are all within the protection scope of the present invention.

Claims

1. A multi-scale chip defect size measurement method based on UNet++, characterized in that: The following steps are involved: Construct a multi-category, multi-scale semantic segmentation dataset of chip surface defect images, use the dataset to train the UNet++ semantic segmentation network, and obtain the UNet++ semantic segmentation model; The chip surface defect image is input into the UNet++ semantic segmentation model. Feature maps of different scales are obtained from the decoder part of the segmentation model. The feature maps of different scales are input into the prediction head based on the attention mechanism. The scale-aware attention module (SAAM) is used to extract multi-scale effective features to obtain a multi-scale semantic segmentation image. The connected domains of each defect category in the multi-scale semantic segmentation image are extracted. For the connected domains of depressions and stains, the contour approximation algorithm is used to annotate the defect contours. For the connected domains of abrasions and scratches, the skeleton extraction algorithm is used to annotate the defect skeletons, thus obtaining a multi-scale defect feature annotated image. The multi-scale defect feature annotation image is input into the multi-scale attention feature fusion module, and multiple feature maps are fused through the multi-feature fusion module MFFM to obtain the fused defect feature annotation image; Extract the contour point set of dents and stains from the fused defect feature annotation image, calculate the perimeter and area of ​​the image pixel level, and extract the skeleton point set of abrasions and scratches to calculate the image pixel level length; Obtain the pixel length corresponding to the known length in the chip surface defect image, calculate the ratio of the actual length to the pixel length, and convert the defect size information at the image pixel level into the actual size information of the image defect.

2. The multi-scale chip defect size measurement method based on UNet++ according to claim 1 is characterized in that: The construction of a multi-category, multi-scale chip surface defect image semantic segmentation dataset includes: Use optical cameras and optical microscopes to collect images of chip surface defects of different sizes and categories; Preprocessing the collected chip surface defect images, including bilateral filtering and histogram equalization; Performing dataset expansion on the preprocessed image to obtain an expanded dataset; The images in the amplified dataset are annotated with pixel-level semantic segmentation and divided into training and test sets to obtain a semantic segmentation dataset of chip surface defect images.

3. The multi-scale chip defect size measurement method based on UNet++ according to claim 1 is characterized in that: Get feature maps of different scales from the decoder part of the segmentation model, including: For the input chip surface defect image, the encoder structure of the UNet++ semantic segmentation model is used to downsample the defect image, extract the features of the defect image, and obtain the encoder feature output; For the encoder feature output, the decoder structure and dense skip connection structure of the UNet++ semantic segmentation model are used to perform feature upsampling and feature fusion to obtain the decoder feature output; Get feature maps of different scales from the decoder part of the segmentation model.

4. The multi-scale chip defect size measurement method based on UNet++ according to claim 3 is characterized in that: The feature maps of different scales are obtained from the decoder part of the segmentation model, specifically the feature maps of the last three layers of the decoder, whose sizes are H×W×64, and The sizes of the semantic segmentation images after the prediction head based on the attention mechanism are: H×W×3, and Where H and W are the vertical and horizontal pixel numbers of the input image, respectively.

5. The multi-scale chip defect size measurement method based on UNet++ according to claim 3 is characterized in that: The structure of the prediction head based on the attention mechanism includes: A scale-aware attention module (SAAM) to extract multi-scale effective features; A 1×1 convolution to adjust the number of channels; A softmax classifier to classify pixels; The structure of the scale-aware attention module SAAM includes: A channel attention module (CAB) to adjust the attention level of each channel; A spatial attention module (SAB) to enhance the module's ability to focus on specific locations; A scale convolution module (SCM) is used to enhance the module’s representation of features and capture multi-scale and multi-resolution context.

6. The multi-scale chip defect size measurement method based on UNet++ according to claim 5, characterized in that: The feature maps of different scales are input into the prediction head based on the attention mechanism to obtain multi-scale semantic segmentation images, including: The feature maps are input into the scale-aware attention module SAAM respectively, and the channel attention module CAB and the spatial attention module SAB are used to enhance the attention to specific channels and spaces, and the scale convolution module SCM is used to enhance the multi-scale representation of features; Input the feature map into 1×1 convolution for feature extraction and adjust the number of channels; The feature map is input into the softmax classifier to classify the pixels and obtain semantic segmentation images of different scales.

7. The multi-scale chip defect size measurement method based on UNet++ according to claim 1, characterized in that: Obtain multi-scale defect feature annotation images, including: Grayscale the semantic segmentation images of different scales to extract the connected domains of various defects; Create blank images of corresponding sizes for semantic segmentation images of different scales respectively; For the connected areas of depressions and stains, the Douglas-Peucker algorithm is used to perform contour approximation processing to obtain the contour point sets of depressions and stains, and mark them on the corresponding blank images; For the connected domains of abrasions and scratches, the Zhang-Suen thinning algorithm is used to extract the skeleton to obtain the skeleton point sets of abrasions and scratches, and annotate them on the corresponding blank images.

8. The multi-scale chip defect size measurement method based on UNet++ according to claim 1, characterized in that: The structure of the multi-scale attention feature fusion module includes: Two upsampling modules to resize feature maps to the same size; A multi-feature fusion module MFFM, used to fuse multiple feature maps; A combination module of two 3×3 convolutions, Batch Normalization, and ReLU activation functions for further feature extraction and channel adjustment; The structure of the multi-feature fusion module MFFM includes: A multi-feature attention gating module (MFAG) that enhances low-level semantic features of the original feature map by upsampling features and uses grouped convolution to reduce computational complexity; A channel attention module CAB and a spatial attention module SAB are used to highlight the important channels and spatial positions of the target area in the feature map and improve the ability to express features.

9. The multi-scale chip defect size measurement method based on UNet++ according to claim 8, characterized in that: The multi-scale feature annotated image is input into the multi-scale attention feature fusion module, including: Resize the feature map through the upsampling module; The feature map is input into the multi-feature fusion module MFFM for feature fusion to obtain the feature map; After two 3×3 convolutions, Batch Normalization and ReLU activation function combination modules, further feature extraction and channel adjustment are performed to obtain the defect feature annotation image.

10. The multi-scale chip defect size measurement method based on UNet++ according to claim 1, characterized in that: The ratio of the actual length to the pixel length L pp for: Where, L pp is the physical size of each pixel, L d is the shooting distance, L f is the focal length of the camera, P c It is the number of pixels corresponding to 1 cm of the camera's light-receiving element.

Images