Defect detection method, defect detection apparatus, and storage medium

Abstract

Provided in the present invention are a defect detection apparatus, a defect detection method, and a computer-readable storage medium. The defect detection method comprises the following steps: acquiring detection images of a plurality of detection units of a sample to be subjected to detection; performing subtraction on a first detection image of a target detection unit and a second detection image of each of a plurality of reference detection units; with each pixel value involved in each pixel point in the first detection image being y, a pixel difference value recorded for a pixel point at each coordinate position corresponding to each pixel value in each difference value image being x, and the number of pixel points having a corresponding pixel difference value in each difference value image being z, separately constructing a plurality of feature maps of the target detection unit; separately performing an expansion operation on the feature maps, and constructing a high-dimensional feature map of the target detection unit on the basis of an intersection of the feature maps; and performing sparse sampling on the high-dimensional feature map, and inputting into a pre-trained defect detection model a defect feature map obtained by means of sparse sampling, so as to determine a detection result of the target detection unit by means of the defect detection model.

Description

Defect detection method, defect detection device and storage medium Technical Field

[0001] The present invention relates to the field of semiconductor detection, and in particular to a defect detection method, a defect detection device, and a computer-readable storage medium. Background Art

[0002] Automatic Optical Inspection (AOI) technology is widely used in semiconductors, LEDs, pan-semiconductors, solar panels and other fields to achieve fast, high-precision and non-destructive defect detection. The semiconductor manufacturing process usually involves a variety of different processes, and wafer defect detection, classification and filtering play an important role in improving the yield of finished products in each process. Existing defect detection methods generally use conventional convolutional neural networks to perform neural network inference on sample images of ordinary sizes to achieve defect detection, classification and filtering. However, this conventional defect detection method has high requirements for defect size. If the defect size accounts for too small a proportion of the training image (for example: less than 10*10 pixels), its feature information will easily be overwhelmed by background information. At this time, it is difficult for the convolutional neural network to effectively extract the feature information indicating the defect, making it difficult to classify and identify these small-sized defects.

[0003] In order to overcome the above-mentioned defects of the prior art, there is an urgent need in the art for an improved defect detection technology for effectively detecting, classifying and filtering small defects to improve the accuracy of defect detection.

[0004] Summary of the Invention

[0005] The following is a brief summary of one or more aspects to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceivable aspects and is neither intended to identify key or critical elements of all aspects nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that will be provided later.

[0006] In order to overcome the above-mentioned defects of the prior art, the present invention provides a defect detection method, a defect detection device and a computer-readable storage medium, which can construct a feature map based on the pixel value difference between the target detection unit and its adjacent reference detection unit, and obtain feature data indicating the defect location by dilation operation and sparse sampling of each feature map, and then perform accurate and effective defect detection, classification and filtering based on a pre-trained defect detection model, thereby improving the accuracy of small defect detection.

[0007] Specifically, the above-mentioned defect detection method provided according to the first aspect of the present invention includes the following steps: obtaining detection images of multiple detection units of the sample to be tested; subtracting the first detection image of the target detection unit from the second detection images of multiple reference detection units adjacent to it to construct multiple difference images of the target detection unit; taking each pixel value involved in each pixel point in the first detection image as y, taking the pixel difference recorded by the pixel points at each coordinate position corresponding to each pixel value in each difference image as x, and taking the number of pixels with corresponding pixel difference values ​​in each difference image as z, respectively constructing multiple feature maps of the target detection unit; performing expansion operations on each feature map, and constructing a high-dimensional feature map of the target detection unit based on their intersection; and sparsely sampling the high-dimensional feature map, and inputting the defect feature map obtained by sparse sampling into a pre-trained defect detection model to determine the detection result of the target detection unit via the defect detection model.

[0008] Furthermore, in some embodiments of the present invention, the first detection image completely covers the target detection unit. Each second detection image completely covers the corresponding reference detection unit. Alternatively, the first detection image covers a first ROI region in the target detection unit. Each second detection image covers a second ROI region at the same position in the corresponding reference detection unit.

[0009] Furthermore, in some embodiments of the present invention, the step of performing an expansion operation on each of the feature maps includes: using a 3*3 convolution kernel to perform a preset number of expansion operations on each of the feature maps to obtain corresponding expanded feature maps.

[0010] Furthermore, in some embodiments of the present invention, the step of sparsely sampling the high-dimensional feature map includes: determining a target sampling number for the sparse sampling based on the feature dimensions of a pre-trained defect detection model; and configuring a higher first sampling frequency for the peripheral area of ​​the high-dimensional feature map and a lower second sampling frequency for the internal area of ​​the high-dimensional feature map based on the target sampling number, so as to perform the sparse sampling on the high-dimensional feature map and obtain a defect feature map that conforms to the feature dimensions of the defect detection model.

[0011] Furthermore, in some embodiments of the present invention, the step of performing the sparse sampling on the high-dimensional feature map and obtaining a plurality of defect features that conform to the feature dimensions of the defect detection model includes: in response to completing the sparse sampling, counting the number of defect feature points obtained by sampling; and in response to the number of defect feature points obtained by sampling being less than the feature dimensions of the defect detection model, filling in 0 for the missing defect features in the peripheral area and / or the internal area according to the first sampling frequency and / or the second sampling frequency to obtain a defect feature map that conforms to the feature dimensions of the defect detection model.

[0012] Furthermore, in some embodiments of the present invention, the step of performing sparse sampling on the high-dimensional feature map and obtaining multiple defect features that conform to the feature dimensions of the defect detection model includes: in response to completing the sparse sampling, normalizing each defect feature point obtained by sampling to obtain a normalized defect feature map that conforms to the feature dimensions of the defect detection model.

[0013] Furthermore, in some embodiments of the present invention, the defect detection model utilizes a deep learning model and includes a feature extraction module and a feature classification module. The feature extraction module includes a feature point selection unit, a convolutional neural network, a first batch normalization unit, and a maximum pooling layer, which is used to extract the multidimensional feature data required for defect detection from multiple input defect features. The feature classification module includes a fully connected layer, a second batch normalization unit, and a discarding unit, which is used to determine the corresponding detection result based on the multidimensional feature data provided by the feature extraction module.

[0014] Furthermore, in some embodiments of the present invention, the input format of the feature point selection unit is B*N*3. Here, B is the number of batches of defect feature maps required for one model training. N is the number of feature points involved in each of the defect feature maps, and each feature point involves three dimensions (x, y, z). The feature point selection unit first selects M feature points on the defect feature map according to the farthest distance, and then selects K feature points as point families of each of the center points with the M feature points as the center points, and then takes the maximum value point or mean value in each of the point families as the output of the point family to obtain output data with an output format of B*M*K*3, and transmits the output data to the convolutional neural network at the back end.

[0015] Furthermore, in some embodiments of the present invention, the feature extraction module includes multiple first blocks. Each of the first blocks is configured with a feature point selection unit, a convolutional neural network, a first batch normalization unit, and a maximum pooling layer. The feature classification module includes multiple second blocks and a first fully connected layer. Each of the second blocks is configured with a second fully connected layer, a second batch normalization unit, and a discarding unit. The first fully connected layer is provided at the back end of each of the second blocks to output the detection results of the defect detection model.

[0016] Furthermore, in some embodiments of the present invention, the step of training the defect detection model includes: obtaining detection image samples of multiple detection units of multiple training samples and their corresponding annotation data. The annotation data indicates whether there is a defect in the corresponding detection image sample; multiple first image samples are selected from the multiple detection image samples of each training sample, and the detection image samples of multiple detection units adjacent to the first image samples are used as second image samples; each first image sample is subtracted from its corresponding multiple second image samples to construct multiple difference images of each first image sample; each pixel value involved in each pixel point in the first image sample is y, the pixel difference recorded by the pixel point at each coordinate position corresponding to each pixel value in each difference image is x, and the number of pixels with corresponding pixel difference values ​​in each difference image is z, respectively, to construct multiple feature maps of each first image sample; each feature map is expanded and a high-dimensional feature map of the first image sample is constructed based on the intersection of the feature maps; the high-dimensional feature map is sparsely sampled, and the defect feature map obtained by the sparse sampling is input into the defect detection model to be trained to obtain the corresponding detection result through the defect detection model; and the learning parameters of the defect detection model are corrected according to the difference between the detection result corresponding to each first image sample and the annotation data to train the defect detection model.

[0017] Furthermore, in some embodiments of the present invention, the step of correcting the learning parameters of the defect detection model based on the difference between the detection results corresponding to each first image sample and the labeled data to train the defect detection model includes: dividing the defect feature map obtained by sparsely sampling the high-dimensional feature map of each first image sample into a training set and a validation set according to a preset ratio; setting the learning rate of the defect detection model to 0.001, the decay rate to 0.0001, and selecting a cross-entropy loss function to train the defect detection model using the training samples in the training set; each time the training samples in the training set are used for a preset number of training, the accuracy of the current defect detection model is verified using the validation samples in the validation set; in response to the result of the accuracy verification reaching a preset accuracy threshold, it is determined that the training of the defect detection model is completed.

[0018] Furthermore, the defect detection apparatus provided according to the second aspect of the present invention includes a memory and a processor. The memory stores computer instructions. The processor is connected to the memory and configured to execute the computer instructions stored in the memory to implement the defect detection method provided according to the first aspect of the present invention.

[0019] Furthermore, in some embodiments of the present invention, the defect detection device further includes a light source, an illumination control unit, an imaging unit, a sensor, and a motion stage. The light source is used to provide illumination light. The illumination control unit is used to transmit the illumination light to the detection unit surface of the sample to be tested to provide bright field illumination thereto. The imaging unit faces the detection unit surface of the sample to be tested. The sensor is used to capture the detection image within the field of view of the imaging unit. The motion stage is used to carry the sample to be tested and to move the plurality of detection units thereof sequentially into the field of view of the imaging unit so that the sensor can capture the corresponding detection images.

[0020] Furthermore, the computer-readable storage medium provided in accordance with the third aspect of the present invention stores computer instructions, which, when executed by a processor, implement the defect detection method provided in accordance with the first aspect of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above features and advantages of the present invention will be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or characteristics may have the same or similar reference numerals.

[0022] FIG1 shows a schematic structural diagram of a defect detection device according to some embodiments of the present invention.

[0023] FIG2 shows a schematic diagram of a sample to be tested according to some embodiments of the present invention.

[0024] FIG3 shows a schematic diagram of an algorithm architecture of a defect detection device according to some embodiments of the present invention.

[0025] FIG4 shows a schematic diagram of a defect detection model provided according to some embodiments of the present invention.

[0026] FIG5 is a schematic flow chart showing a training phase of a defect detection method according to some embodiments of the present invention.

[0027] FIG6 shows a schematic diagram of detecting image samples according to some embodiments of the present invention.

[0028] FIG7 shows a schematic diagram of a feature map provided in an XY plane according to some embodiments of the present invention.

[0029] FIG8 is a schematic flow chart showing a detection phase of a defect detection method according to some embodiments of the present invention. DETAILED DESCRIPTION

[0030] The following specific embodiments illustrate the embodiments of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Although the description of the present invention will be introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to this embodiment. On the contrary, the purpose of introducing the invention in conjunction with the embodiment is to cover other options or modifications that may be extended based on the claims of the present invention. In order to provide a deep understanding of the present invention, the following description will include many specific details. The present invention can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present invention, some specific details will be omitted in the description.

[0031] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0032] Furthermore, the terms "upper," "lower," "left," "right," "top," "bottom," "horizontal," and "vertical" used in the following description should be understood to refer to the orientations depicted in that section and the accompanying drawings. These relative terms are used solely for convenience of description and do not necessarily imply that the devices described herein must be manufactured or operated in a specific orientation. Therefore, they should not be construed as limiting the present invention.

[0033] It will be understood that although the terms "first," "second," "third," etc. may be used herein to describe various components, regions, layers, and / or portions, these components, regions, layers, and / or portions should not be limited by these terms, and these terms are merely used to distinguish different components, regions, layers, and / or portions. Thus, a first component, region, layer, and / or portion discussed below may be referred to as a second component, region, layer, and / or portion without departing from some embodiments of the present invention.

[0034] As mentioned above, existing defect detection methods generally use conventional convolutional neural networks to perform neural network inference on sample images of standard sizes to achieve defect detection, classification, and filtering. However, this conventional defect detection method has high requirements for defect size. If the defect size is too small in proportion to the training image, its feature information will easily be overwhelmed by the background information. In this case, the convolutional neural network will have difficulty effectively extracting the feature information indicating the defect, making it difficult to classify and identify these small defects.

[0035] In order to overcome the above-mentioned defects of the prior art, the present invention provides a defect detection method, a defect detection device and a computer-readable storage medium, which can construct a feature map based on the pixel value difference between the target detection unit and its adjacent reference detection unit, and obtain feature data indicating the defect location by dilation operation and sparse sampling of each feature map, and then perform accurate and effective defect detection, classification and filtering based on a pre-trained defect detection model, thereby improving the accuracy of small defect detection.

[0036] In some non-limiting embodiments, the defect detection method provided in the first aspect of the present invention can be implemented based on the defect detection device provided in the second aspect of the present invention. Specifically, the defect detection device provided in the first aspect of the present invention may include a memory and a processor. The memory includes, but is not limited to, the computer-readable storage medium provided in the third aspect of the present invention, which stores computer instructions. The processor is connected to the memory and configured to execute the computer instructions stored in the memory to implement the defect detection method provided in the first aspect of the present invention.

[0037] Further, please refer to Figure 1. Figure 1 shows a schematic structural diagram of a defect detection device provided according to some embodiments of the present invention.

[0038] In the embodiment shown in Figure 1, the above-mentioned defect detection device provided by the first aspect of the present invention may include a detection module 11, a light source 12, a lighting control unit 13, an imaging unit 14, a sensor 15 and a motion stage 16. Here, the above-mentioned memory and processor can be configured in the detection module 11. The light source 12 is used to provide illumination light. The lighting control unit 13 is used to transmit the illumination light provided by the light source 12 to the detection unit surface of the sample to be tested 17 to provide bright field illumination thereto. The imaging unit 14 faces the detection unit surface of the sample to be tested 17. The sensor 15 is used to collect detection images within the field of view of the imaging unit 14. The motion stage 16 is used to carry the sample to be tested 17 and move its multiple detection units in sequence into the field of view of the imaging unit 14 so that the sensor 15 can collect corresponding detection images.

[0039] Please further refer to FIG2 , which shows a schematic diagram of a sample to be tested according to some embodiments of the present invention.

[0040] In the embodiment shown in FIG2 , the sample to be tested 17 may include multiple dies. The defect detection device may select the entire die or part of the die in the sample to be tested 17 as a detection unit based on the size of the die and the field of view of the imaging unit 14, and use the detection images of the target detection unit and at least one reference detection unit adjacent thereto as a set of test data. For example, for the detection scenario of small-sized defects (e.g., less than 10*10 pixels), the defect detection device may select a first ROI (Region of Interest) area 1712 in the target detection unit to obtain a first detection image, and select second ROI areas 1711 and 1713 with the same relative position in two reference detection units adjacent thereto to obtain a second detection image, and then use the first detection image of the first ROI area 1712 and the second detection images of the second ROI areas 1711 and 1713 to form a set of test data 171.

[0041] In addition, please refer to Figure 3. Figure 3 shows a schematic diagram of an algorithm architecture of a defect detection device provided according to some embodiments of the present invention.

[0042] In the embodiment of FIG3 , the defect detection device provided by the present invention involves an offline model training module 31 and an online defect detection module 32. The model training module 31 includes multiple sets of detection image samples 311 with labeled data, an image feature high-dimensional mapping unit 312, and a defect detection model 313 to be trained. The model training module 31 is configured to determine the defect feature map corresponding to the input detection image sample 311 via the image feature high-dimensional mapping unit 312, and train the defect detection model 313 based on the defect feature map and its corresponding labeled data. Correspondingly, the defect detection module 32 includes a similar image feature high-dimensional mapping module 322 and a trained defect detection model 323. The model training module 311 is configured to obtain detection images 321 of multiple detection units of the test sample 17, determine the corresponding defect feature map InputPointMap via the image feature high-dimensional mapping module 322, and then determine the defect detection results corresponding to the defect feature map via the trained defect detection model 323.

[0043] Furthermore, in some embodiments, the defect detection model 313 may utilize a deep learning model such as ResNet or VGG, which includes a feature extraction module and a feature classification module. The feature extraction module is configured to extract multidimensional feature data required for defect detection from multiple input defect features. The feature classification module is configured to determine the corresponding detection result based on the multidimensional feature data provided by the feature extraction module.

[0044] Please refer to FIG4 for details, which shows a schematic structural diagram of a defect detection model provided according to some embodiments of the present invention.

[0045] In the embodiment shown in Figure 4, the feature extraction module may include two first blocks 41. Here, each first block 41 is configured with a feature point selection unit PointSelect, a convolutional neural network (CNN), a batch normalization unit (BN) and a maximum pooling layer Maxpool. The feature point selection unit PointSelect is used to select feature map points from the input defect feature map InputPointMap. The convolutional neural network CNN is arranged after the feature point selection unit PointSelect, and is used to extract valid feature data based on the feature map points and activation function (for example: Relu) selected by it. The batch normalization unit BN is arranged after the convolutional neural network CNN, and is used to normalize the valid feature data extracted by it. The maximum pooling layer Maxpool is arranged after the batch normalization unit BN, and is used to downsample the valid feature data extracted by it.

[0046] In addition, the feature classification module includes two second blocks 421 and a first fully connected layer 422. Here, each second block 421 is configured with a second fully connected layer (Linear), a second batch normalization unit (BN) and a dropout unit. The second fully connected layer Linear is used to classify multidimensional feature data. The second batch normalization unit BN is used to normalize the classified multidimensional feature data. The dropout unit Dropout is used to randomly drop neurons to prevent overfitting. The first fully connected layer 422 is provided at the back end of each second block 421, and is used to further classify the multidimensional feature data that has been classified, normalized and randomly dropped neurons to output the detection results of the defect detection model 313.

[0047] The following describes the working principle of the above-mentioned defect detection device in conjunction with some examples of defect detection methods. Those skilled in the art will understand that these examples of defect detection methods are merely some non-limiting implementation methods provided by the present invention, intended to clearly demonstrate the main concepts of the present invention and provide some specific solutions that are convenient for the public to implement, rather than limiting the full functions or working methods of the defect detection device. Similarly, the defect detection device is merely a non-limiting implementation method provided by the present invention and does not constitute a limitation on the execution subject and execution order of the various steps in these defect detection methods.

[0048] In the non-limiting embodiment shown in FIG3 , the defect detection method provided by the first aspect of the present invention can be implemented in two phases: offline training and online testing. During the offline training phase, a technician can first train a defect detection model 313 using a model training module 31 and multiple sets of inspection image samples 311 with labeled data. Then, during the online testing phase, the trained defect detection model 323 can be used to determine the inspection results of each inspection unit of the sample 17 to be tested.

[0049] Please refer to Figures 1, 5 and 6 for details. Figure 5 shows a flow chart of the training phase of the defect detection method provided according to some embodiments of the present invention, and Figure 6 shows a schematic diagram of detection image samples provided according to some embodiments of the present invention.

[0050] As shown in Figures 1 and 5, during the training of the defect detection model 313, the model training module 31 can first use the motion stage 16 to sequentially move the multiple detection units of multiple training samples into the field of view of the imaging unit 14, allowing the sensor 15 to capture the corresponding detection image samples and obtain the corresponding annotation data. Here, the annotation data indicates whether the corresponding detection image samples contain defects and can be obtained through traditional image algorithms or manual recognition.

[0051] Afterwards, as shown in FIG6 , the model training module 31 can select a plurality of first image samples TestImg from each detection image sample, and use the detection image samples of a plurality of (for example, two) detection units adjacent thereto as second image samples RefImg1 and RefImg2 to construct a plurality of groups of detection image samples 311. Here, the first image sample TestImg can completely cover the corresponding detection unit, or only cover the first ROI area 1712 in the corresponding detection unit, where the defect is represented by an abnormal brightness signal. Correspondingly, each second image sample RefImg1 and RefImg2 can also completely cover the corresponding detection unit, or respectively cover the second ROI area 1711 and 1713 at the same position in the corresponding detection unit, where defects with abnormal brightness signals may or may not exist depending on the actual situation.

[0052] Furthermore, in the process of selecting the plurality of first image samples TestImg, the number of first image samples TestImg with defects and first image samples TestImg without defects may preferably be kept in a roughly equal ratio to improve the credibility of the training result.

[0053] Thereafter, the model training module 31 may perform high-dimensional mapping on each feature data in the plurality of groups of detection image samples 311 via its image feature high-dimensional mapping unit 312 to highlight the defect features.

[0054] Specifically, during the high-dimensional mapping process, the image feature high-dimensional mapping unit 312 can first perform a difference between each first image sample TestImg and its corresponding multiple second image samples RefImg1 and RefImg2 to construct multiple difference images DeltaImg1 and DeltaImg2 for each first image sample TestImg. Here, DeltaImg1 records the difference in pixel value of each pixel between the first image sample TestImg and the second image sample RefImg1. DeltaImg2 records the difference in pixel value of each pixel between the first image sample TestImg and the second image sample RefImg2.

[0055] Next, please refer to Figure 7. Figure 7 shows a schematic diagram of a feature map provided in an XY plane according to some embodiments of the present invention.

[0056] As shown in Figure 7, the image feature high-dimensional mapping unit 312 can use the pixel values ​​involved in each pixel point in the first image sample TestImg as y, the pixel difference recorded by the pixel points at each coordinate position corresponding to each pixel value in each difference image DeltaImg1 and DeltaImg2 as x, and the number of pixel points with corresponding pixel difference values ​​in each difference image DeltaImg1 and DeltaImg2 as z to construct multiple feature maps FeatureMap1 and FeatureMap2 for each first image sample TestImg, and characterize the defects in the corresponding first image sample TestImg through the feature points (x0, y0, z0) with abnormal coordinates in each feature map FeatureMap1 and FeatureMap2.

[0057] For example, for a first image sample TestImg involving multiple pixel values ​​such as {100, 101, 102, ..., 130, 131}, the image feature high-dimensional mapping unit 312 can select any pixel value (e.g., 100) in the first image sample TestImg and determine the corresponding multiple pixel positions. Subsequently, since the pixels in the second image sample RefImg1 correspond one-to-one with the pixels in the first image sample TestImg, the image feature high-dimensional mapping unit 312 can determine the pixel value array {99, 100, 101, 100, 102} involved in each corresponding pixel position in the second image sample RefImg1, and subtract each element in the array from the aforementioned pixel value (i.e., 100) to obtain the corresponding difference value array {-1, 0, 1, 0, 2}. Afterwards, the image feature high-dimensional mapping unit 312 can use the pixel value (i.e., 100) as y, the pixel differences recorded in the difference array as x, and the number of pixel differences in the difference array as z to determine the coordinates (x0, y0, z0) of a feature point in the feature map FeatureMap1, and then traverse the pixel values ​​involved in the first image sample TestImg as described above to obtain the feature map FeatureMap1 shown in Figure 7.

[0058] In this way, by converting the brightness signal features on the first image sample TestImg into position features in the form of (x, y, z), the present invention can effectively increase the detection sensitivity of the defect detection model 313 to low-contrast signals and avoid the characteristic signals of small-size defects being overwhelmed by the large-size image background information.

[0059] Afterwards, please continue to refer to Figure 5. The image feature high-dimensional mapping unit 312 can first use a 3*3 convolution kernel to perform a preset number of expansion operations (for example, 3 times) on each feature map FeatureMap1 and FeatureMap2 to obtain the corresponding expanded feature maps, and then construct and output the high-dimensional feature map of the first image sample TestImg based on the intersection of the two. Afterwards, the image feature high-dimensional mapping unit 312 can sparsely sample the high-dimensional feature map according to the input feature dimension of the defect detection model 313 to obtain a defect feature map InputPointMap that meets the input feature dimension of the defect detection model 313, and construct a model sample data set based on the defect feature map InputPointMap of each detection image sample and its corresponding annotation data.

[0060] Furthermore, in view of the actual situation that the defect feature points are mainly distributed in the periphery of the high-dimensional feature map, the image feature high-dimensional mapping unit 312 can configure a higher first sampling frequency for the peripheral area of ​​the high-dimensional feature map and a lower second sampling frequency for the internal area of ​​the high-dimensional feature map, so as to perform the above-mentioned sparse sampling on the high-dimensional feature map, thereby retaining the defect feature points as much as possible to improve the defect detection rate.

[0061] Furthermore, for the case where the number of feature points in some high-dimensional feature maps is insufficient, in order to avoid inconsistent dimensions of feature information obtained by sampling, the image feature high-dimensional mapping unit 312 can also preferably fill in 0 for defect features missing in the peripheral area and / or internal area, and normalize the defect features obtained by sampling to obtain a defect feature map InputPointMap with consistent feature dimensions of size N*3, where N is the number of feature points in the map, and each feature point involves three coordinate dimensions of (x, y, z).

[0062] Afterwards, the model training module 31 can divide the above model sample data set into a training set, a validation set, and a test set according to a preset ratio, and input the defect feature map InputPointMap in the training set and the validation set into the defect detection model 313 to be trained in sequence to obtain the corresponding detection result output value.

[0063] Specifically, as shown in Figure 4, the input format of the defect detection model 313 can be B*N*3, where B (e.g., B=16) is the number of batches of defect feature maps required for one model training run, and N is the number of feature points in each defect feature map InputPointMap, where each feature point is represented by three coordinate dimensions (x, y, z). During model inference, the defect detection model 313 can first obtain B*N*3 dimensional input data from the training set and input it into the feature point selection unit PointSelect of the feature extraction module.

[0064] After that, in response to obtaining the above-mentioned input data of the B*N*3 dimension from the image feature high-dimensional mapping unit 312, the feature point selection unit PointSelect can first select M (M < N) feature points on the defect feature map according to the farthest distance, and then use these M feature points as the center points to respectively select K feature points as the point families of each center point, and then take the maximum value point or the mean value in each point family as the output of the point family to obtain output data with the output format of B*M*K*3, and transmit the output data to the convolutional neural network CNN at the back end. After that, in response to obtaining the above-mentioned feature map point data of the B*M*K*3 dimension from the feature point selection unit PointSelect, the convolutional neural network CNN can extract effective feature data based on these selected feature map points and an activation function (e.g., Relu), and transmit the effective feature data to the first batch normalization unit BN and the max pooling layer Maxpool at the back end. After that, in response to obtaining the above-mentioned effective feature data from the convolutional neural network CNN, the first batch normalization unit BN and the max pooling layer Maxpool can sequentially perform normalization and downsampling processing on the extracted effective feature data to convert the effective feature data into multi-dimensional feature data required by the feature classification module at the back end.

[0065] After that, in response to the multi-dimensional feature data obtained from the feature extraction module, the feature classification module can first perform classification, normalization, and random neuron dropout processing on the multi-dimensional feature data through the second fully connected layer, the second batch normalization unit (BN), and the dropout unit configured in its second block 421, and then further classify the multi-dimensional feature data that has undergone the classification, normalization, and random neuron dropout processing through the first fully connected layer 422 to determine the detection result output value of the defect detection model 313.

[0066] After that, the model training module 31 can, according to the preset learning rate, decay rate, and loss function, correct each learning parameter in the defect detection model 313 according to the difference between the detection result output value corresponding to each first image sample TestImg and the annotation data (i.e., the true value) to obtain a trained defect detection model 323.

[0067] Specifically, in the process of training the above-mentioned defect detection model 313, the model training module 31 can set its learning rate to 0.001, the decay rate to 0.0001, and select the cross-entropy loss function to use the training samples in the training set to train the defect detection model 313. Furthermore, in some embodiments, the model training module 31 can use the verification samples in the verification set to perform an accuracy verification on the current defect detection model 114 every time it uses the training samples in the training set for a preset number of training times (for example, 4 times). In response to the result of the accuracy verification reaching a preset accuracy threshold (for example, 98%), or the training error between the training sample and the verification sample gradually stabilizes near the preset training error, the model training module 31 can determine that the training of the defect detection model 313 is completed.

[0068] The following will continue to describe the relevant steps of the online detection phase of the defect detection method. Please refer to Figures 1 to 4 and Figures 6 to 8. Figure 8 shows a flow chart of the detection phase of the defect detection method provided according to some embodiments of the present invention.

[0069] As shown in Figures 1 to 3 and 8, during the process of defect detection of the sample to be tested 17, the defect detection module 32 of the defect detection device can first move the multiple detection units of the sample to be tested 17 into the field of view of the imaging unit 14 in sequence via the motion stage 16, so that the sensor 15 can respectively collect the detection images 321 of each detection unit, and then construct a corresponding detection data group based on the first detection image TestImg of each target detection unit and the second detection images RefImg1 and RefImg2 of the multiple reference detection units adjacent thereto. Here, the first detection image TestImg can completely cover the target detection unit, or it can only cover the first ROI area 1712 in the target detection unit. Correspondingly, each second detection image RefImg1 and RefImg2 can also completely cover the corresponding reference detection unit, or respectively cover the second ROI areas 1711 and 1713 at the same position in the corresponding reference detection unit.

[0070] Afterwards, based on the detection data groups of the above-mentioned target detection units, the defect detection module 32 can highlight the defect features in the detection image 321 of the target detection unit through the image feature high-dimensional mapping unit 322 to avoid the characteristic signals of small-size defects being overwhelmed by the large-size image background information.

[0071] Specifically, in the process of highlighting defect features in the detection image 321 of the target detection unit, the image feature high-dimensional mapping unit 322 can first perform a difference between the first detection image TestImg of the target detection unit and the second detection images RefImg1 and RefImg2 of multiple adjacent reference detection units to construct multiple difference images DeltaImg1 and DeltaImg2 of the target detection unit. Here, the difference image DeltaImg1 records the difference in pixel value of each pixel between the first detection image TestImg and the second detection image RefImg1, and the difference image DeltaImg2 records the difference in pixel value of each pixel between the first detection image TestImg and the second detection image RefImg2.

[0072] Afterwards, the image feature high-dimensional mapping unit 322 can construct multiple feature maps FeatureMap1 and FeatureMap2 of the target detection unit, respectively, using the pixel values ​​involved in each pixel point in the first detection image TestImg as y, the pixel difference values ​​recorded at the pixel points at each coordinate position corresponding to each pixel value in each difference image DeltaImg1 and DeltaImg2 as x, and the number of pixels with corresponding pixel difference values ​​in each difference image as z, and characterize the defects in the first detection image TestImg with the abnormal position coordinates (x0, y0, z0) of the feature points as shown in Figure 7. In this way, by converting the brightness signal features shown in Figure 6 on the first image sample TestImg into position features in the form of (x, y, z) as shown in Figure 7, the present invention can effectively increase the detection sensitivity of the defect detection model 323 to low-contrast signals and prevent the feature signals of small-sized defects from being overwhelmed by the large-sized image background information.

[0073] Afterwards, the image feature high-dimensional mapping unit 322 can first use a 3*3 convolution kernel as described above to perform a preset number of expansion operations (for example, 3 times) on each feature map FeatureMap1 and FeatureMap2 to obtain the corresponding expanded feature maps, and then construct and output the high-dimensional feature map of the target detection unit based on their intersection. Afterwards, the image feature high-dimensional mapping unit 322 can also determine the target sampling number (for example, N=256) based on the input feature dimension of the pre-trained defect detection model 323, and then sparsely sample the high-dimensional feature map based on the target sampling number to obtain a defect feature map InputPointMap that meets the input feature dimension of the defect detection model 323.

[0074] Furthermore, in view of the actual situation that the defect feature points are mainly distributed in the periphery of the high-dimensional feature map, the image feature high-dimensional mapping unit 322 can configure a higher first sampling frequency for the peripheral area of ​​the high-dimensional feature map and a lower second sampling frequency for its internal area as described above, and then sparsely sample the high-dimensional feature map accordingly to retain the defect feature points as much as possible to improve the defect detection rate.

[0075] Furthermore, in response to the completion of sparse sampling, the image feature high-dimensional mapping unit 322 may also preferably count the number of defect feature points obtained by sampling. In response to the number of defect feature points obtained by sampling being less than the input feature dimension of the defect detection model 323, the image feature high-dimensional mapping unit 322 may also fill in the missing defect features in the peripheral area and / or the internal area with zeros according to the first sampling frequency and / or the second sampling frequency, to obtain a defect feature map InputPointMap that meets the input feature dimension of the defect detection model 323.

[0076] In addition, in response to the completion of sparse sampling, the image feature high-dimensional mapping unit 322 may also preferably perform normalization processing on each defect feature point obtained by sampling to obtain a normalized defect feature map having a feature dimension that conforms to the defect detection model.

[0077] Afterwards, as shown in Figures 3, 4 and 8, the image feature high-dimensional mapping unit 322 can input the defect feature map InputPointMap that meets the input feature dimension of the defect detection model 323 into the pre-trained defect detection model 323 to determine the detection result of the target detection unit through the defect detection model 323.

[0078] Specifically, the input format of the defect detection model 323 can be N*3, where N is the number of feature points involved in each defect feature map InputPointMap, and each feature point involves three coordinate dimensions (x, y, z).

[0079] During the model inference process, the defect detection model 313 can first input the defect feature map InputPointMap into the feature point selection unit PointSelect of the feature extraction module. In response to obtaining the above-mentioned N*3 dimensional input data from the image feature high-dimensional mapping unit 322, the feature point selection unit PointSelect can first select M (M≤N) feature points on the defect feature map according to the farthest distance, and then select K feature points as point families of each center point with the M feature points as the center points, and then take the maximum value point or mean in each point family as the output of the point family to obtain output data with an output format of M*K*3, and transmit the output data to the back-end convolutional neural network CNN. Afterwards, in response to obtaining the above-mentioned M*K*3 dimensional feature map point data from the feature point selection unit PointSelect, the convolutional neural network CNN can extract valid feature data based on these selected feature map points and activation functions (for example: Relu), and transmit the valid feature data to the first batch of normalization units BN and maximum pooling layer Maxpool at the back-end. Afterwards, in response to obtaining the above-mentioned valid feature data from the convolutional neural network CNN, the first batch of normalization units BN and the maximum pooling layer Maxpool can normalize and downsample the extracted valid feature data in turn to convert the valid feature data into the multi-dimensional feature data required by the back-end feature classification module.

[0080] Thereafter, in response to the multidimensional feature data obtained from the feature extraction module, the feature classification module can first classify, normalize and randomly drop neurons on the multidimensional feature data through the second fully connected layer, the second batch normalization unit (BN) and the dropout unit configured in its second block 421, and then further classify the multidimensional feature data processed by the classification, normalization and random drop neurons through the first fully connected layer 422 to determine the detection result output value of the defect detection model 323, and determine whether there is a defect in the corresponding target detection unit based on the detection result output value.

[0081] In summary, the above-mentioned defect detection device, defect detection method and computer-readable storage medium provided by the present invention can construct a feature map based on the pixel value difference between the target detection unit and its adjacent reference detection unit, and obtain feature data indicating the defect location through expansion operations and sparse sampling of each feature map, and then perform accurate and effective defect detection, classification and filtering based on the pre-trained defect detection model, thereby improving the accuracy of small defect detection.

[0082] Although the above methods are illustrated and described as a series of acts for simplicity of explanation, it is to be understood and appreciated that these methods are not limited by the order of the acts, as some acts may occur in a different order and / or concurrently with other acts from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art according to one or more embodiments.

[0083] Those skilled in the art will appreciate that information, signals, and data may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips cited throughout the foregoing description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0084] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. A skilled person may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0085] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A defect detection method, characterized in that, Including the following steps: Obtain detection images of multiple detection units of a sample to be measured; Subtract the first detection image of a target detection unit from the second detection images of multiple adjacent reference detection units respectively to construct multiple difference images of the target detection unit; Taking each pixel value involved in each pixel point in the first detection image as y, taking the pixel differences recorded by the pixel points at the corresponding coordinate positions of each pixel value in each of the difference images as x, and taking the number of pixel points with corresponding pixel differences in each of the difference images as z, respectively construct multiple feature maps of the target detection unit; Perform dilation operations on each of the feature maps respectively, and construct a high-dimensional feature map of the target detection unit according to their intersection; And Perform sparse sampling on the high-dimensional feature map, and input the defect feature map obtained by the sparse sampling into a pre-trained defect detection model to determine the detection result of the target detection unit via the defect detection model.

2. The defect detection method according to claim 1, wherein The first detection image completely covers the target detection unit, and each of the second detection images completely covers the corresponding reference detection unit, or The first detection image covers the first ROI area in the target detection unit, and each of the second detection images covers the second ROI area at the same position in the corresponding reference detection unit.

3. The defect detection method according to claim 1, wherein The step of performing dilation operations on each of the feature maps respectively includes: Use a 3*3 convolution kernel to perform dilation operations on each of the feature maps for a preset number of times to obtain corresponding dilated feature maps respectively.

4. The defect detection method according to claim 1, characterized in that, The step of performing sparse sampling on the high-dimensional feature map includes: Determine the target sampling number of the sparse sampling according to the feature dimension of the pre-trained defect detection model; and According to the target sampling number, configure a higher first sampling frequency for the peripheral area of the high-dimensional feature map and a lower second sampling frequency for the internal area of the high-dimensional feature map to perform the sparse sampling on the high-dimensional feature map and obtain a defect feature map that conforms to the feature dimension of the defect detection model.

5. The defect detection method according to claim 4, wherein The step of performing the sparse sampling on the high-dimensional feature map and obtaining multiple defect features that conform to the feature dimension of the defect detection model includes: In response to completing the sparse sampling, count the number of defect feature points obtained by the sampling; and In response to the number of defect feature points obtained by the sampling being less than the feature dimension of the defect detection model, fill in 0 for the missing defect features in the peripheral area and / or the internal area according to the first sampling frequency and / or the second sampling frequency to obtain a defect feature map that conforms to the feature dimension of the defect detection model.

6. The defect detection method according to claim 4, wherein The step of performing the sparse sampling on the high-dimensional feature map and obtaining multiple defect features that conform to the feature dimension of the defect detection model includes: In response to completing the sparse sampling, perform normalization processing on each defect feature point obtained by the sampling to obtain a normalized defect feature map that conforms to the feature dimension of the defect detection model.

7. The defect detection method according to claim 1, characterized in that, The defect detection model selects a deep learning model and includes a feature extraction module and a feature classification module, where The feature extraction module includes a feature point selection unit, a convolutional neural network, a first batch normalization unit, and a max pooling layer, and is used to extract multi-dimensional feature data required for defect detection from multiple input defect features. The feature classification module includes a fully connected layer, a second batch normalization unit, and a dropout unit, and is used to determine the corresponding detection result according to the multi-dimensional feature data provided by the feature extraction module.

8. The defect detection method according to claim 7, wherein The input format of the feature point selection unit is B*N*3, where B is the number of batches of defect feature maps required for one model training, N is the number of feature points involved in each defect feature map, and each of the feature points involves 3 dimensions of (x, y, z). The feature point selection unit first selects M feature points on the defect feature map according to the farthest distance, then selects K feature points as the point families of each of the M feature points as the center points, and then takes the maximum value point or the mean value of each point family as the output of the point family to obtain output data with the output format of B*M*K*3, and transmits the output data to the convolutional neural network at the back end.

9. The defect detection method according to claim 7, characterized in that The feature extraction module includes a plurality of first blocks, and each of the first blocks is configured with a feature point selection unit, a convolutional neural network, a first batch normalization unit, and a max pooling layer. The feature classification module includes a plurality of second blocks and a first fully connected layer. Each of the second blocks is configured with a second fully connected layer, a second batch normalization unit, and a dropout unit. The first fully connected layer is arranged at the back end of each of the second blocks to output the detection result of the defect detection model.

10. The defect detection method according to claim 7, characterized in that, The steps of training the defect detection model include: Obtaining detection image samples of multiple detection units of multiple training samples and their corresponding annotation data, where the annotation data indicates whether there are defects in the corresponding detection image samples. Respectively selecting multiple first image samples from the multiple detection image samples of each training sample, and using the detection image samples of multiple adjacent detection units as second image samples. Respectively taking the difference between each first image sample and its corresponding multiple second image samples to construct multiple difference images of each first image sample. Taking each pixel value involved in each pixel point in the first image sample as y, taking the pixel difference recorded by the pixel points at the corresponding coordinate positions of each pixel value in each difference image as x, and taking the number of pixel points with corresponding pixel differences in each difference image as z, and respectively constructing multiple feature maps of each first image sample. Respectively performing dilation operations on each of the feature maps, and constructing a high-dimensional feature map of the first image sample according to their intersection. Performing sparse sampling on the high-dimensional feature map, and inputting the defect feature map obtained by sparse sampling into the defect detection model to be trained, so as to obtain the corresponding detection result through the defect detection model; and According to the difference between the detection result corresponding to each first image sample and the annotation data, correcting the learning parameters of the defect detection model to train the defect detection model.

11. The defect detection method according to claim 10, characterized in that, The step of correcting the learning parameters of the defect detection model according to the differences between the detection results corresponding to each of the first image samples and the annotation data to train the defect detection model includes: Dividing the defect feature map obtained by sparsely sampling the high-dimensional feature maps of each of the first image samples into a training set and a validation set according to a preset ratio; Setting the learning rate of the defect detection model to 0.001, the decay rate to 0.0001, and selecting a cross-entropy loss function to train the defect detection model using the training samples in the training set; Every time the training samples in the training set are used for a preset number of trainings, the accuracy of the current defect detection model is verified once using the validation samples in the validation set; In response to the result of the accuracy verification reaching a preset accuracy threshold, it is determined that the training of the defect detection model is completed.

12. A defect detection device, characterized in that, including: A memory on which computer instructions are stored; and A processor connected to the memory and configured to execute the computer instructions stored on the memory to implement the defect detection method according to any one of claims 1 to 11.

13. The defect detection device according to claim 12, characterized in that, It further includes: A light source for providing illumination light; An illumination control unit for transmitting the illumination light to the surface of the detection unit of the sample to be measured to provide bright-field illumination thereto; An imaging unit facing the surface of the detection unit of the sample to be measured; A sensor for collecting detection images within the field of view of the imaging unit; and A moving stage for carrying the sample to be measured and sequentially moving its multiple detection units into the field of view of the imaging unit for the sensor to collect corresponding detection images.

14. A computer-readable storage medium having computer instructions stored thereon, characterized in that, When the computer instructions are executed by the processor, the defect detection method according to any one of claims 1 to 11 is implemented.

Images