Defect detection method

Abstract

The present disclosure relates to a defect detection method. The defect detection method comprises: acquiring a first image, the first image being an image of a sample to be detected for defects; inputting the first image into a pre-trained semantic segmentation network to obtain a second image; determining a gradient image of the first image as a third image, performing neighborhood gradient statistics based on the third image to obtain a fourth image; fusing the second image and the fourth image to obtain a fifth image; and performing threshold segmentation on pixel values in the fifth image to determine a defect region. The defect detection method of the present disclosure has high accuracy and high sensitivity.

Description

Technical Field

[0001] This disclosure relates to the field of defect detection, and more particularly to a defect detection method. Background Technology

[0002] With the continuous advancement of semiconductor manufacturing technology and the rapid expansion of integrated circuit scale, ensuring high quality and high yield in chip production has become a critical step. Defective regions in images can be detected using scanning electron microscopy (SEM) or optical microscopy (OM), and these defective regions represent defects in the chip sample.

[0003] For detecting defects in images, defect detection algorithms are typically used. These algorithms are broadly categorized into two types: traditional image processing algorithms and deep learning algorithms. Traditional image processing algorithms utilize edge detection and morphological operations to detect defect areas. Deep learning algorithms, employing networks such as Convolutional Neural Networks (CNNs), learn from large amounts of labeled data to detect defects in images. Traditional image processing algorithms offer high sensitivity; however, when images contain numerous features, they struggle to accurately distinguish between defective and normal regions, resulting in low accuracy in defect detection. Deep learning algorithms offer high accuracy; however, when images contain minute defects, they are prone to missing these small defects, leading to lower overall defect detection sensitivity.

[0004] Therefore, the accuracy and sensitivity of the defect detection methods involved in the relevant technologies cannot be simultaneously achieved, resulting in either low accuracy or low sensitivity. Summary of the Invention

[0005] To overcome the problems existing in related technologies, this disclosure provides a defect detection method.

[0006] This disclosure provides a defect detection method, the method comprising: acquiring a first image, the first image being an image of a sample to be defect-detected; inputting the first image into a pre-trained semantic segmentation network to obtain a second image, wherein the input of the semantic segmentation network is an image, and the output of the semantic segmentation network is a defect probability image, wherein the pixel values ​​in the defect probability image are within a range defined by a preset first value and a preset second value; determining a gradient image of the first image as a third image, performing neighborhood gradient statistics based on the third image to obtain a fourth image; fusing the second image and the fourth image to obtain a fifth image; and performing threshold segmentation on the pixel values ​​in the fifth image to determine defect regions.

[0007] In some embodiments, the step of performing neighborhood gradient statistics based on the third image to obtain a fourth image includes: performing neighborhood gradient statistics on the third image and normalizing the result of the neighborhood gradient statistics to a value range defined by the preset first value and the preset second value to obtain a fourth image.

[0008] In some embodiments, performing neighborhood gradient statistics on the third image includes: for each pixel in the third image, calculating the sum of pixel values ​​of pixels within a preset size adjacent to each pixel, and obtaining the sum value corresponding to each pixel as the result of neighborhood gradient statistics.

[0009] In some embodiments, the step of calculating the sum of pixel values ​​of pixels within a preset size adjacent to each pixel in the third image to obtain the summation value corresponding to each pixel includes: performing convolution with the third image based on a convolution kernel having the preset size to obtain the summation value corresponding to each pixel in the third image.

[0010] In some embodiments, determining the gradient image of the first image includes: calculating the gradient value of each pixel in the first image to obtain the gradient image of the first image.

[0011] In some embodiments, obtaining the first image includes: obtaining the image size of an initial image to be detected and obtaining an input size, wherein the input size is an image size suitable for input to the pre-trained semantic segmentation network; comparing the image size and the input size; if the image size of the initial image is different from the input size, adjusting the image size of the initial image so that the adjusted image size is the same as the input size, and using the adjusted initial image as the first image; if the image size of the initial image is the same as the input size, then using the initial image as the first image.

[0012] In some embodiments, the semantic segmentation network is pre-trained as follows: multiple first training images containing defective regions are acquired; the defective regions in the first training images are labeled to obtain a first mask image, wherein the first mask image is a binarized image, and the pixel values ​​of the corresponding defective regions in the first mask image are preset first values, and the pixel values ​​of the non-defective regions are preset second values, wherein the first value is greater than the second value; the semantic segmentation network is trained based on the first training images and the corresponding first mask images to obtain the pre-trained semantic segmentation network.

[0013] In some embodiments, fusing the second image and the fourth image to obtain a fifth image includes: fusing the second image and the fourth image based on a pre-trained image fusion network to obtain a fifth image.

[0014] In some embodiments, the image fusion network is pre-trained as follows: Multiple second training images containing defective regions are acquired; a third defect probability image and a fourth defect probability image corresponding to the second training images are acquired, wherein the third defect probability image is a defect probability image output by the pre-trained semantic segmentation network, and the fourth defect probability image is a defect probability image obtained by normalizing the result of neighborhood gradient statistics to a value range defined by the preset first value and the preset second value; defective regions of the second training images are labeled to obtain a second mask image, wherein the second mask image is a binarized image, and the pixel value corresponding to the defective region in the second mask image is the preset first value, and the pixel value of the non-defective region is the preset second value, where the first value is greater than the second value; the image fusion network is trained based on the third defect probability image, the fourth defect probability image, and the second mask image corresponding to the second training images to obtain the pre-trained image fusion network.

[0015] In some embodiments, thresholding the pixel values ​​in the fifth image to determine defect regions includes: identifying image regions in the fifth image whose pixel values ​​are greater than or equal to a preset pixel threshold as defect regions.

[0016] This disclosure provides a defect detection method that acquires a first image of a sample to be detected. Based on the first image, a defect probability image is obtained using a pre-trained semantic segmentation network and used as a second image. The second image retains the high accuracy advantage of deep learning algorithms. Based on the first image, a corresponding gradient image is acquired, and neighborhood gradient statistics are performed on the gradient image to obtain a fourth image. The fourth image retains the high sensitivity advantage of traditional image processing algorithms. The second and fourth images are fused to obtain a fifth image. The defect region is determined based on the fifth image as the defect detection result. The fifth image integrates information from both the semantic and gradient features of the first image. This defect detection method balances accuracy and sensitivity, exhibiting high accuracy and high sensitivity. Compared to defect detection methods based on deep learning algorithms, it improves the sensitivity of defect detection methods; compared to defect detection methods based on traditional image processing algorithms, it improves the accuracy of defect detection methods.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0019] Figure 1 This is a flowchart illustrating a defect detection method according to an exemplary embodiment.

[0020] Figure 2 This is a flowchart illustrating pre-training a semantic segmentation network according to an exemplary embodiment.

[0021] Figure 3 This is a schematic diagram illustrating pre-training a semantic segmentation network according to an exemplary embodiment.

[0022] Figure 4 This is a flowchart illustrating an example of acquiring a first image according to an exemplary embodiment.

[0023] Figure 5 This is a schematic diagram illustrating a method for obtaining a defect probability image using a pre-trained semantic segmentation network, according to an exemplary embodiment.

[0024] Figure 6 This is a flowchart illustrating neighborhood gradient statistics for a gradient image according to an exemplary embodiment.

[0025] Figure 7 This is a flowchart illustrating another method for obtaining a defect probability image, according to an exemplary embodiment.

[0026] Figure 8a This is an SEM image of a sample to be tested, as illustrated in an exemplary embodiment.

[0027] Figure 8b This is a gradient image illustrated according to an exemplary embodiment.

[0028] Figure 8c This is a defect probability image illustrated according to an exemplary embodiment.

[0029] Figure 9a This is an SEM image of a sample to be tested, as illustrated in an exemplary embodiment.

[0030] Figure 9b This is a gradient image illustrated according to an exemplary embodiment.

[0031] Figure 9c This is a defect probability image illustrated according to an exemplary embodiment.

[0032] Figure 10 This is a flowchart illustrating a fusion process to obtain a fifth image according to an exemplary embodiment.

[0033] Figure 11 This is a flowchart illustrating the pre-training of an image fusion network according to an exemplary embodiment.

[0034] Figure 12 This is a flowchart illustrating the determination of a defect region in a fifth image according to an exemplary embodiment.

[0035] Figure 13 This is a schematic diagram illustrating a defect detection method according to an exemplary embodiment. Detailed Implementation

[0036] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure.

[0037] The defect detection method provided in this disclosure is applied in the field of defect detection, for example, it can be applied to the scenario of defect detection of images of samples to be tested in the semiconductor field.

[0038] In related technologies, traditional image processing algorithms, due to their high sensitivity, can detect minute defect areas during defect detection. However, for images with complex features, traditional image processing algorithms struggle to accurately distinguish defective areas from normal areas, leading to false positives or false negatives and resulting in low accuracy in detecting defective areas. Deep learning algorithms can be used to detect defects in images with complex features. However, when using deep learning algorithms to process images containing minute defective areas, multi-level pooling and dimensionality reduction operations can easily dilute or lose the features of these minute defective areas, causing them to be missed and resulting in low sensitivity in defect detection.

[0039] In view of this, embodiments of the present disclosure provide a defect detection method, comprising: acquiring a first image to be detected; performing semantic segmentation on the first image to obtain a second image; and performing neighborhood gradient statistical processing on the gradient image of the first image to obtain a fourth image; fusing the second image and the fourth image to obtain a fifth image that integrates semantic and gradient features; and performing threshold segmentation on the fifth image to distinguish defective and non-defective regions according to a set threshold, thereby locating the defective region in the first image. The defect detection method exhibits high accuracy and sensitivity.

[0040] Figure 1 This is a flowchart illustrating a defect detection method according to an exemplary embodiment. Figure 1 As shown, it includes steps S11-S15.

[0041] Step S11: Obtain the first image.

[0042] The first image is an image of the sample to be inspected for defects, and the first image includes the defect area.

[0043] In this embodiment of the disclosure, the first image can be understood as an optical microscope (OM) image or a scanning electron microscope (SEM) image obtained on the sample to be inspected for defects. The sample can be understood as a wafer or a photomask. This disclosure does not limit the image of the sample to be inspected for defects.

[0044] In this embodiment of the disclosure, the defect in the image is an image of the defect in the sample, and the defect in the image can be referred to as the defect region.

[0045] Step S12: Input the first image into the pre-trained semantic segmentation network to obtain the second image.

[0046] In this embodiment of the disclosure, the first image is input into a pre-trained semantic segmentation network to obtain a defect probability image, and the obtained defect probability image is used as the second image.

[0047] The semantic segmentation network takes an image as input and outputs a defect probability image. The semantic segmentation network can be a deep learning-based semantic segmentation network. Semantic segmentation networks are existing technologies; for example, they can be U-Net networks or other convolutional neural networks used for image segmentation tasks, such as DeepLab networks. The pixel values ​​in the defect probability image output by the semantic segmentation network fall within a range defined by a preset first value and a preset second value.

[0048] Step S13: Determine the gradient image of the first image as the third image, and perform neighborhood gradient statistics based on the third image to obtain the fourth image.

[0049] In some examples, for the first image, the gradient value of each pixel in the first image is determined using a gradient calculation algorithm to obtain a gradient image, which includes the gradient value. The gradient image is then used as the third image, and neighborhood gradient statistics are performed on the third image to obtain the fourth image.

[0050] This disclosure does not limit the choice of gradient calculation algorithm.

[0051] In one example, the gradient images of the first image in the horizontal and vertical directions are calculated according to the Sobel operator, and the gradient image of the first image is obtained based on the gradient images in the horizontal and vertical directions.

[0052] In one example, the gradient images of the first image in the horizontal and vertical directions can be calculated based on the Scharr operator or the Laplacian operator, and the gradient image of the first image can be obtained based on the gradient images in the horizontal and vertical directions.

[0053] In one example, the formula is used. Calculate the gradient image of the first image, or use the formula G = |G| x |+|G y The gradient image of the first image is calculated. Here, G is the gradient value of the first image at a certain location. x G represents the gradient value of the horizontal gradient image at this location. y The gradient value of the gradient image at this location is the vertical gradient value, the horizontal gradient value is the horizontal direction of the first image, and the vertical gradient value is the vertical direction of the first image.

[0054] In this embodiment, semantic segmentation is performed on the first image to obtain the second image, which helps to detect defect regions in the first image that include large defect regions. However, the detection capability for the first image that includes small defect regions is limited, and it is easy to miss small defect regions. Moreover, the size of the defect region in the first image is usually unknown information. In order to make the defect detection method of this disclosure applicable to the detection of the first image that includes small defect regions, step S13 needs to be performed to obtain the fourth image by performing neighborhood gradient statistics based on the gradient image of the first image (i.e., the third image), so as to detect the small defect regions in the first image and improve the accuracy of the defect detection method.

[0055] In the fourth image, the range of pixel values ​​is defined by a preset first value and a preset second value.

[0056] For ease of explanation, this disclosure uses steps S12 and S13 to describe the process of obtaining the second image from the first image and the process of obtaining the fourth image from the first image, respectively. However, it should be understood that this disclosure does not limit the order in which steps S12 and S13 are executed.

[0057] Step S14: Fuse the second image and the fourth image to obtain the fifth image.

[0058] In this embodiment of the disclosure, the second image output by the pre-trained semantic segmentation network is fused with the fourth image obtained by neighborhood gradient statistics to obtain the fifth image.

[0059] For example, the fusion method can use weighted average or other fusion algorithms.

[0060] Step S15: Threshold segmentation is performed on the pixel values ​​in the fifth image to determine the defect area.

[0061] In this embodiment of the disclosure, a threshold is set to distinguish the pixel values ​​in the fifth image, thereby determining the defect region. Specifically, determining the defect region means determining the location of the defect region. Since the first, second, third, fourth, and fifth images have the same image size, the location of the defect region determined in the fifth image is the same as the location of the defect region determined in the first image.

[0062] According to an exemplary embodiment of this disclosure, a second image is obtained by inputting a first image into a pre-trained semantic segmentation network. The semantic segmentation network is then used to predict the probability that each pixel in the first image belongs to a defect, resulting in a defect probability image, which serves as the second image. Simultaneously, a gradient image (which can also be understood as a third image) of the first image is determined, and neighborhood gradient statistics are performed to obtain a fourth image. Gradient information and neighborhood statistics are used to highlight local feature changes in the image. Image information is obtained from two different dimensions: semantic segmentation and image gradient. The second and fourth images are then fused to obtain a fifth image. The second image retains the high accuracy advantage of deep learning algorithms, the fourth image retains the high sensitivity advantage of traditional image processing algorithms, and the fifth image integrates information from both semantic and detail features, balancing the sensitivity and accuracy of defect detection. This allows for a balance between accuracy and sensitivity in subsequent threshold segmentation to determine the defect region, resulting in a defect detection method with high accuracy and high sensitivity. Compared to defect detection methods based on deep learning algorithms, this method improves sensitivity; compared to defect detection methods based on traditional image processing algorithms, it improves accuracy.

[0063] In this embodiment of the disclosure, the following method will be used to target Figure 1 The steps involved are further explained below.

[0064] In this embodiment of the disclosure, the first image is semantically segmented to obtain the second image.

[0065] In one example, a pre-trained (referred to as pre-trained) semantic segmentation network is used to perform semantic segmentation on the first image to obtain the second image.

[0066] Among them, the following are adopted Figure 2 The method shown is used to obtain a pre-trained semantic segmentation network.

[0067] Figure 2 This is a flowchart illustrating pre-training a semantic segmentation network according to an exemplary embodiment. For example... Figure 2 As shown, it includes steps S21-S23.

[0068] Step S21: Obtain multiple first training images containing defective regions.

[0069] The first training image can be understood as an OM image or SEM image containing defective regions, and the image size of the first training image is suitable for input into the semantic segmentation network. The image source of the first training image can cover various scenarios where defects may occur, and this disclosure does not limit it.

[0070] Step S22: Mark the defect regions in the first training image to obtain the first mask image.

[0071] In this embodiment of the disclosure, defective regions in each first training image are labeled to form a corresponding first mask image.

[0072] The first mask image is a binarized image, and the pixel values ​​corresponding to defective regions in the first mask image are preset first values, while the pixel values ​​of non-defective regions are preset second values, where the first value is greater than the second value. The first mask image can also be understood as the first label during the training of the semantic segmentation network.

[0073] In one example, when the range of values ​​for the binarized image is set to 0 to 1, the preset first value can be 1, and the preset second value can be 0.

[0074] In another example, when the range of values ​​for the binarized image is set to 0 to 255, the preset first value can be 255, and the preset second value can be 0.

[0075] Step S23: Based on the first training image and the corresponding first mask image, train the semantic segmentation network to obtain the pre-trained semantic segmentation network.

[0076] In this embodiment of the disclosure, a semantic segmentation network is trained based on multiple first training images and corresponding first mask images to obtain a pre-trained semantic segmentation network.

[0077] The output of the pre-trained semantic segmentation network is a defect probability image, in which the value corresponding to each pixel is used to characterize the probability that the pixel is a defect.

[0078] In one example, when the value range of the binarized image is set to 0 to 1, the second image obtained using a pre-trained semantic segmentation network has values ​​ranging from [0,1]. For instance, if a pixel in the second image has a value of 0.75, then the probability of that pixel being a defect is 75%.

[0079] In another example, when the value range of the binarized image is set to 0 to 255, the second image obtained using a pre-trained semantic segmentation network has values ​​ranging from [0, 255]. For example, if the pixel value of a certain pixel in the second image is 204, then the probability that the pixel represents a defect is... That is, the probability that the pixel is a defect is 80%.

[0080] According to an exemplary embodiment of this disclosure, by pre-training a semantic segmentation network, a first mask image is set as a binarized image, and the pixel values ​​of defective and non-defective regions are clearly defined.

[0081] Figure 3This is a schematic diagram illustrating pre-training a semantic segmentation network according to an exemplary embodiment.

[0082] like Figure 3 As shown in this embodiment, a large number of first training images containing defective regions are acquired. For each first training image, the defective regions are manually labeled to generate a corresponding first mask image. The first mask image is essentially a binary image, with pixel values ​​containing two values. For example, if the value range of the binary image is set to 0 to 255, in the first mask image, the pixel value representing the defective region is set to 255, while the pixel value representing the non-defective region is set to 0. Alternatively, 0 and 1 can be used to assign values ​​to the pixel values ​​in the first mask, with the same meaning as described above.

[0083] In this embodiment, after the annotation of all first training images is completed, the semantic segmentation network is trained using all first training images and their corresponding first mask images. For example, the semantic segmentation network can be a U-Net network, or other network architectures can be selected according to actual needs. During the training process, the semantic segmentation network continuously learns the correspondence between features and defect regions in the first training images, and gradually improves the segmentation accuracy of defect regions by adjusting the parameters of the semantic segmentation network. After multiple rounds of training and optimization, a pre-trained semantic segmentation network is finally obtained. The input of this pre-trained semantic segmentation network is an image, and the output is a defect probability image.

[0084] In this embodiment of the disclosure, based on the pre-trained semantic segmentation network described above, the first image is input into the pre-trained semantic segmentation network for processing to obtain the second image.

[0085] Figure 4 This is a flowchart illustrating an embodiment of acquiring a first image. For example... Figure 4 As shown, it includes steps S31-S34.

[0086] Step S31: Obtain the image size of the initial image to be detected, and obtain the input size.

[0087] The initial image is the image to be detected, which can be an OM image or a SEM image. The initial image includes the defect region, which corresponds to the defect in the sample.

[0088] The input size is the image size suitable for inputting into the pre-trained semantic segmentation network. The input size is the image size of the first training image and also the image size of the second training image.

[0089] Step S32: Compare the image size with the input size.

[0090] Step S33: If the image size of the initial image is different from the input size, adjust the image size of the initial image so that the image size of the adjusted initial image is the same as the input size, and use the adjusted initial image as the first image.

[0091] In this embodiment of the disclosure, the image size and the input size are compared. If the image size of the initial image is different from the input size, the image size of the initial image is normalized and adjusted so that the adjusted image size is suitable for input into the pre-trained semantic segmentation network, and the adjusted initial image is used as the first image.

[0092] In one example, when the image size of the initial image is different from the input size, zero-padding is applied to the image size of the initial image so that the image size of the zero-padding initial image is the same as the input size.

[0093] In another example, when the initial image size differs from the input size, the initial image is cropped so that the cropped initial image size is the same as the input size.

[0094] Step S34: If the image size of the initial image is the same as the input size, then the initial image is used as the first image.

[0095] According to an exemplary embodiment of this disclosure, since the initial image may have a different size than the first training image, the image size of the initial image is adjusted to fit the input requirements of the pre-trained semantic segmentation network. This ensures that the pre-trained semantic segmentation network can handle initial images of different sizes, improving the adaptability of the semantic segmentation network to different image sizes. Furthermore, by adapting the initial image size before inputting it into the pre-trained semantic segmentation network, errors introduced by image size mismatch are avoided.

[0096] Figure 5 This is a schematic diagram illustrating a method for obtaining a defect probability image using a pre-trained semantic segmentation network, according to an exemplary embodiment.

[0097] In this embodiment of the disclosure, the image size of an initial image is first obtained, and then compared with the input size suitable for input into a pre-trained semantic segmentation network. If the image size of the initial image differs from the input size, the image size of the initial image is adjusted to obtain an image size suitable for input into the pre-trained semantic segmentation network, which is then used as the first image and input into the pre-trained semantic segmentation network to obtain a defect probability image, which is then used as the second image.

[0098] In this embodiment of the disclosure, in addition to obtaining the second image from the first image as described above, it also includes the process of performing gradient processing on the first image to obtain a gradient image, and using the gradient image to obtain a fourth image.

[0099] Figure 6 This is a flowchart illustrating neighborhood gradient statistics for a gradient image according to an exemplary embodiment. Figure 6 As shown, it includes steps S41-S42.

[0100] Step S41: Obtain the third image.

[0101] In this embodiment of the disclosure, the third image is the gradient image.

[0102] The gradient image can be obtained in the manner shown in step S13, which will not be described in detail here.

[0103] Step S42: For each pixel in the third image, the sum of pixel values ​​of the pixels within a preset size adjacent to each pixel is calculated to obtain the sum value corresponding to each pixel, which is used as the result of neighborhood gradient statistics.

[0104] In this embodiment of the disclosure, in the third image, for each pixel, the neighboring pixels in its adjacent region are counted. For example, they may be the 4 neighboring pixels directly adjacent to it, or the 8 neighboring pixels diagonally adjacent to it, etc. The sum of the pixel values ​​corresponding to the neighboring pixels is counted to obtain the sum of the pixel values ​​in the region adjacent to the pixel, and the sum is used as the neighborhood gradient statistics result of the pixel.

[0105] The sum of pixel values ​​of pixels within a preset size adjacent to each pixel can be understood as, for example, using a 9*9 matrix of all ones as the convolution kernel to perform convolution processing on the third image.

[0106] In some examples, for each pixel in the third image, the sum of the pixel values ​​of the pixels within a preset size adjacent to each pixel is calculated to obtain the sum value corresponding to each pixel, including:

[0107] The third image is convolved with a convolution kernel of a preset size to obtain the summation value corresponding to each pixel in the third image.

[0108] In some examples, the preset size is the size of the convolution kernel. The size and value of the convolution kernel can be determined as needed. For example, the width and height of the convolution kernel are both greater than or equal to 3, and the values ​​are, for example, both 1.

[0109] In some examples, the edges of the third image are padded, and the padded third image is then convolved to obtain the results of neighborhood gradient statistics. Padded edges of the image before convolving it with a convolution kernel is a prior art technique, and those skilled in the art can reasonably set the required padded size based on the size of the convolution kernel.

[0110] The edges of the third image are filled, for example, with zero filling, symmetrical filling, or reflection filling. This disclosure does not limit the filling method.

[0111] According to an exemplary embodiment of this disclosure, in a defect detection scenario, since defects often cause changes in the features of surrounding pixels, by summing the neighboring pixel values ​​of each pixel, the local change features caused by the defect are integrated, so that the defect area and the normal area show obvious differences in statistical results, thereby making it easier to accurately locate the defect and improve the accuracy of defect detection.

[0112] Figure 7 This is a flowchart illustrating another method for obtaining a defect probability image, according to an exemplary embodiment. For example... Figure 7 As shown, steps S51-S52 are included.

[0113] Step S51: Obtain the third image.

[0114] In this embodiment of the disclosure, the third image is obtained as shown in step S41, which will not be described in detail here.

[0115] Step S52: Perform neighborhood gradient statistics on the third image, and normalize the results of the neighborhood gradient statistics to a range defined by a preset first value and a preset second value to obtain the fourth image.

[0116] In this embodiment of the disclosure, for example, the method shown in step S42 is used to perform neighborhood gradient statistics on the third image, and the result of the neighborhood gradient statistics is normalized to a range of values ​​defined by a preset first value and a preset second value to obtain a defect probability image after neighborhood gradient statistics, and the defect probability image obtained after neighborhood gradient statistics is used as the fourth image.

[0117] The preset first value and the preset second value can be understood in the manner shown in step S22, which will not be elaborated here.

[0118] According to an exemplary embodiment of this disclosure, the neighborhood gradient statistics results are normalized to a specific value range, so that the data of the third image after neighborhood gradient statistics has a uniform scale, which helps to accurately detect defect regions in the image. When fusing with the second image output by the semantic segmentation network, using a uniform value range can make the fusion operation smoother and more stable, and improve the fusion effect.

[0119] In this embodiment of the disclosure, a picture is obtained as shown... Figure 8a The image shown is a first image with a regular blocky arrangement but also containing obvious defective areas. Figure 8a The first image shown can be understood as the SEM image of the sample corresponding to the defect detection in step S11.

[0120] In this embodiment of the disclosure, a gradient calculation algorithm is used to calculate, such as Figure 8a The gradient value of each pixel in the first image shown is used to highlight the details of the defective areas in the first image, thereby obtaining an image containing obvious defective areas, such as... Figure 8b The gradient image shown is a gradient image (which can also be understood as a third image).

[0121] In this embodiment of the disclosure, for each pixel in the third image, the sum of pixel values ​​in its preset-size neighborhood (e.g., a 9*9 neighborhood) is calculated. The calculated sum of pixel values ​​is then normalized to a range between a preset first value and a preset second value, thereby obtaining... Figure 8c The defect probability image shown is used as the fourth image.

[0122] In this embodiment of the disclosure, a picture is obtained as shown... Figure 9a The first image shown includes a region of minute defects. Figure 9a The first image shown can be understood as an SEM image of the sample to be inspected for defects, as in step S11.

[0123] In this embodiment of the disclosure, a gradient calculation algorithm is used to obtain the gradient image corresponding to the first image (wherein, the gradient image can also be understood as the third image). Because Figure 9a There are tiny defect areas in the material. If deep learning algorithms are used for defect detection, it is easy to miss these tiny defect areas. Figure 9b In the gradient image shown, the gradient value of the tiny defect region is significant relative to the gradient value of the surrounding region.

[0124] In this embodiment of the disclosure, by means of such Figure 9b The third image shown is subjected to neighborhood gradient statistics, resulting in the following: Figure 9c The defect probability image shown is used as the fourth image. Among them, in Figure 9cIn the fourth image, the regions with larger values ​​correspond to the regions with large local gradient changes in the first image, which may be defect regions in the first image. These small defect regions are easily diluted or lost during multi-level pooling and dimensionality reduction when using deep learning-based detection methods (e.g., defect detection methods based on semantic segmentation networks), leading to missed detections. Figure 9c In the fourth image shown, the neighborhood gradient statistics of the tiny defect region are significant compared to the neighborhood gradient statistics of the surrounding region.

[0125] According to an exemplary embodiment of this disclosure, gradient information of local regions in an image can be integrated by summing the pixel values ​​in the neighborhood of each pixel in the third image. In defect detection, since defects often cause changes in the gradient of surrounding pixels, using neighborhood statistics can amplify the features of these local gradient changes, making the defective region more prominent in the statistical results. This helps to detect defects more accurately and improves the accuracy of defect detection.

[0126] The second and fourth images obtained above are then fused to produce the fifth image.

[0127] Figure 10 This is a flowchart illustrating a fusion process to obtain a fifth image according to an exemplary embodiment. For example... Figure 10 As shown, steps S61-S62 are included.

[0128] Step S61: Obtain the second and fourth images.

[0129] In this embodiment of the disclosure, the following methods can be used: Figure 4 The second image is obtained in the manner shown, using the method described above. Figure 7 The fourth image is obtained in the manner shown, which will not be elaborated here.

[0130] Step S62: Based on the pre-trained image fusion network, fuse the second image and the fourth image to obtain the fifth image.

[0131] In this embodiment of the disclosure, the second image and the fourth image are fused according to a pre-trained image fusion network to obtain the fifth image.

[0132] In one example, the second and fourth images are fused using pre-trained Deeply-Fused Nets to obtain the fifth image.

[0133] In order to achieve image fusion, the second and fourth images have the same image size; in addition, since the fourth image is obtained by gradient processing of the first image, the image sizes of the third and fourth images are the same as the image size of the first image when performing defect detection.

[0134] According to an exemplary embodiment of this disclosure, since the second image obtained by defect detection based on a semantic segmentation network has the advantages of high accuracy and high robustness, and the gradient-based defect detection algorithm has the advantage of high sensitivity, the second image and the fourth image are fused to obtain a fused image that incorporates the advantages of both semantic segmentation network processing and neighborhood gradient statistical processing. The fifth image thus balances both the sensitivity and accuracy of defect detection.

[0135] Among them, the following are adopted Figure 11 The pre-trained image fusion network is obtained in the manner shown.

[0136] Figure 11 This is a flowchart illustrating the pre-training of an image fusion network according to an exemplary embodiment. For example... Figure 11 As shown, it includes steps S71-S74.

[0137] Step S71: Obtain multiple second training images containing defective regions.

[0138] In this embodiment, the second training image can be understood as an OM image or SEM image containing defective regions, and the image size of the second training image is suitable for input into the semantic segmentation network. The image source of the second training image can cover various scenarios where defects may occur, and this disclosure does not limit it. The second training image can be the same as the first training image.

[0139] Step S72: Obtain the third and fourth defect probability images corresponding to the second training image.

[0140] The third defect probability image is the defect probability image output by a semantic segmentation network pre-trained based on the second training image input, and the fourth defect probability image is the defect probability image obtained by performing neighborhood gradient statistics based on the second training image and normalizing the results of the neighborhood gradient statistics to a range defined by a preset first value and a preset second value.

[0141] Step S73: Mark the defect areas of the second training image to obtain the second mask image.

[0142] The second mask image is a binarized image, and the pixel values ​​of the corresponding defect areas in the second mask image are preset first values, while the pixel values ​​of the non-defect areas are preset second values, with the first value being greater than the second value.

[0143] Step S74: Based on the third defect probability image, the fourth defect probability image, and the second mask image corresponding to the second training image, train the image fusion network to obtain the pre-trained image fusion network.

[0144] According to an exemplary embodiment of this disclosure, an image fusion network is used to combine the advantages of semantic segmentation networks and neighborhood gradient statistics to enhance the expression and extraction of defect features in the first image, thereby improving the accuracy and reliability of defect detection.

[0145] In this embodiment of the disclosure, the fused fifth image is processed to obtain the defect region.

[0146] Figure 12 This is a flowchart illustrating the process of determining a defect region in a fifth image according to an exemplary embodiment. For example... Figure 12 As shown, steps S81-S82 are included.

[0147] Step S81: Obtain the fifth image.

[0148] Step S82: The image region in the fifth image whose pixel value is greater than or equal to a preset pixel threshold is identified as a defect region.

[0149] In some embodiments, the fifth image is thresholded, and image regions in the fifth image that are greater than or equal to a preset pixel threshold are determined as defect regions.

[0150] In one example, when the value range of the binarized image is set to 0 to 1, the pixel value range in the fifth image is [0,1]. In this case, the preset pixel threshold is, for example, 0.5, and the image area in the fifth image with a value greater than or equal to 0.5 is identified as a defect area.

[0151] In another example, when the range of values ​​for the binarized image is set to 0 to 255, the range of pixel values ​​in the fifth image is [0, 255]. In this case, the preset pixel threshold is, for example, 128, and the image regions in the fifth image with values ​​greater than or equal to 128 are identified as defect regions.

[0152] In this embodiment of the disclosure, the following is adopted: Figure 13 The embodiments described above are further illustrated in the manner shown.

[0153] Figure 13 This is a schematic diagram illustrating a defect detection method according to an exemplary embodiment.

[0154] like Figure 13As shown, a first image is acquired, and then subjected to defect detection based on a semantic segmentation network and a gradient-based defect detection method. Specifically, in the semantic segmentation network-based defect detection branch, the first image is input into a pre-trained semantic segmentation network to obtain a second image. In the gradient-based defect detection branch, gradient image calculation is performed on the first image to obtain a third image. Neighborhood gradient statistics are performed on the third image, and the results are normalized to obtain a fourth image. The second and fourth images obtained from the different branches are then fused to obtain a fifth image. Thresholding segmentation is performed on the fifth image to identify defect regions.

[0155] In some embodiments, if the image size of the initial image is different from the input size, after thresholding the pixel values ​​in the fifth image to determine the defect region, the defect detection method further determines the location of the defect region in the initial image based on the location of the defect region in the fifth image and the ratio between the image size of the initial image and the input size. Thresholding the pixel values ​​in the fifth image to determine the defect region may also include: thresholding the fifth image, marking image regions with probability values ​​greater than or equal to a preset pixel threshold as defect regions, marking image regions with probability values ​​less than the preset threshold as non-defect regions, and using the marked image as the sixth image; obtaining the bounding box (referred to as the defect bounding box) of the defect region in the sixth image; and using the defect bounding box to indicate the defect region in the sixth image.

[0156] In this disclosure, although operations are described in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0157] The methods and apparatus disclosed herein can be implemented using standard programming techniques, utilizing rule-based logic or other logic to implement various method steps. It should also be noted that the terms "apparatus" and "module" as used herein and in the claims are intended to include implementations using one or more lines of software code and / or hardware implementations and / or devices for receiving input.

[0158] Any step, operation, or procedure described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product comprising a computer-readable medium containing computer program code, which is executable by a computer processor to perform any or all of the described steps, operations, or procedures.

[0159] The foregoing description of embodiments of this disclosure has been provided for purposes of illustration and description. The foregoing description is not exhaustive and is not intended to limit this disclosure to the exact form disclosed; various modifications and variations may be made in accordance with the foregoing teachings, or may be derived from practice of this disclosure. These embodiments were chosen and described to illustrate the principles of this disclosure and its practical application, enabling those skilled in the art to utilize this disclosure in various implementations and modifications suitable for the particular purpose conceived.

[0160] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0161] It is understood that in this disclosure, "multiple" refers to two or more, and other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0162] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0163] It can be further understood that, unless otherwise specified, "connection" includes both direct connections where no other components exist between the two parties and indirect connections where other components exist between them.

[0164] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0165] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. It is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0166] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A defect detection method, characterized in that, The method includes: Acquire a first image, which is an image of the sample to be inspected for defects; The first image is input into a pre-trained semantic segmentation network to obtain a second image, wherein the input of the semantic segmentation network is an image, and the output of the semantic segmentation network is a defect probability image, wherein the range of pixel values ​​in the defect probability image is a range defined by a preset first value and a preset second value. The gradient image of the first image is determined and used as the third image. Neighborhood gradient statistics are performed based on the third image to obtain the fourth image. The second image and the fourth image are merged to obtain the fifth image; Threshold segmentation is performed on the pixel values ​​in the fifth image to determine the defect area.

2. The method according to claim 1, characterized in that, The step of performing neighborhood gradient statistics based on the third image to obtain the fourth image includes: The third image is subjected to neighborhood gradient statistics, and the results of the neighborhood gradient statistics are normalized to the range of values ​​defined by the preset first value and the preset second value to obtain the fourth image.

3. The method according to claim 2, characterized in that, The neighbor gradient statistics for the third image include: For each pixel in the third image, the sum of the pixel values ​​of the pixels within a preset size adjacent to each pixel is calculated to obtain the sum value corresponding to each pixel, which is used as the result of neighborhood gradient statistics.

4. The method according to claim 3, characterized in that, For each pixel in the third image, the summation of pixel values ​​of pixels within a preset size adjacent to each pixel is calculated to obtain the summation value corresponding to each pixel, including: The third image is convolved with a convolution kernel of the preset size to obtain the summation value corresponding to each pixel in the third image.

5. The method according to claim 1, characterized in that, Determining the gradient image of the first image includes: The gradient value of each pixel in the first image is calculated to obtain the gradient image of the first image.

6. The method according to claim 1, characterized in that, The acquisition of the first image includes: Obtain the image size of the initial image to be detected, and obtain the input size, wherein the input size is the image size suitable for input into the pre-trained semantic segmentation network; Compare the image size with the input size; If the image size of the initial image is different from the input size, then the image size of the initial image is adjusted so that the image size of the adjusted initial image is the same as the input size, and the adjusted initial image is used as the first image; If the image size of the initial image is the same as the input size, then the initial image is used as the first image.

7. The method according to claim 1, characterized in that, The semantic segmentation network is pre-trained using the following method: Acquire multiple first training images containing defective regions; The defect regions in the first training image are labeled to obtain a first mask image, wherein the first mask image is a binarized image, and the pixel values ​​corresponding to the defect regions in the first mask image are preset first values, and the pixel values ​​of the non-defect regions are preset second values, wherein the first value is greater than the second value; Based on the first training image and the corresponding first mask image, a semantic segmentation network is trained to obtain a pre-trained semantic segmentation network.

8. The method according to claim 1, characterized in that, The process of fusing the second image and the fourth image to obtain the fifth image includes: The second image and the fourth image are fused together using a pre-trained image fusion network to obtain the fifth image.

9. The method according to claim 8, characterized in that, The image fusion network is pre-trained using the following method: Acquire multiple second training images containing defective regions; Obtain a third defect probability image and a fourth defect probability image corresponding to the second training image, wherein the third defect probability image is a defect probability image output by the pre-trained semantic segmentation network, and the fourth defect probability image is a defect probability image obtained by normalizing the result of neighborhood gradient statistics to a value range defined by the preset first value and the preset second value. The defect regions of the second training image are labeled to obtain a second mask image, wherein the second mask image is a binarized image, and the pixel values ​​of the corresponding defect regions in the second mask image are the preset first value, and the pixel values ​​of the non-defect regions are the preset second value, wherein the first value is greater than the second value; Based on the third defect probability image, the fourth defect probability image, and the second mask image corresponding to the second training image, an image fusion network is trained to obtain a pre-trained image fusion network.

10. The method according to claim 1, characterized in that, The step of thresholding the pixel values ​​in the fifth image to determine the defect region includes: The image regions in the fifth image whose pixel values ​​are greater than or equal to a preset pixel threshold are identified as defect regions.

Images