Automatic optical detection method and device and storage medium

Abstract

The invention discloses an automatic optical detection method and device and a storage medium. The method comprises the steps that original overlook image data of a wafer substrate are collected under irradiation of a first light source; acquiring multi-frame side illumination image data of the wafer substrate under the illumination of a plurality of second light sources in sequence; adjusting the brightness of the original overlook image data according to the multi-frame side-view image data to obtain target overlook image data; loading a preset defect classification network; inputting the target overlook image data into a first feature extraction structure to extract multi-stage first target defect features; inputting the multi-frame side illumination image data into a second feature extraction structure to jointly extract a second target defect feature; inputting the multi-level first target defect features and the second target defect features into a feature fusion structure to be fused into third target defect features; and respectively inputting the third target defect characteristics into a plurality of classification head structures to divide the types of the physical defects existing in the wafer substrate. And the defect detection accuracy of the wafer substrate is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to an automatic optical detection method, device and storage medium. Background Art

[0002] With the rapid development of electronic products such as high-power power supplies and motors, electronic products are shifting towards miniaturization and high power density, which puts higher requirements on the packaging of wafer substrates.

[0003] Wafer substrates are usually surface mounted using processes such as TOLL (TO-Leadless). Before packaging, automated optical inspection (AOI) equipment is used to detect whether the wafer substrate has physical defects such as bumps, breakage, cracks, pores, foreign matter, contamination, etc., to ensure product quality.

[0004] At present, deep learning is gradually used in AOI equipment to detect whether there are defects in wafer substrates. It uses a large number of samples with physical defects to train neural networks and learn the ability to detect defects.

[0005] However, in the actual production process, the probability of defects in wafer substrates is low, resulting in a small number of samples with physical defects, causing overfitting of the neural network, resulting in low performance of the neural network and reducing the accuracy of wafer substrate defect detection. Summary of the invention

[0006] In view of this, the present invention provides an automatic optical inspection method, device and storage medium to improve the accuracy of detecting physical defects on wafer substrates.

[0007] A first aspect of the present invention provides an automatic optical detection method, comprising:

[0008] Collecting original top-down image data of a wafer substrate under the illumination of a first light source; the angle formed between the illumination direction of the first light source and the wafer substrate is a right angle;

[0009] sequentially collecting multiple frames of side illumination image data of the wafer substrate under the illumination of multiple second light sources; the angles formed between the illumination directions of each of the second light sources and the wafer substrate are all acute angles;

[0010] Adjusting the brightness of the original overhead image data according to the multiple frames of the side-illuminated image data to obtain target overhead image data;

[0011] Loading a preset defect classification network; the defect classification network includes a first feature extraction structure, a second feature extraction structure, a feature fusion structure and a plurality of classification head structures;

[0012] Inputting the target overhead image data into the first feature extraction structure to extract a multi-level first target defect feature representing whether there is a physical defect;

[0013] Inputting the plurality of frames of the side illumination image data into the second feature extraction structure to jointly extract a second target defect feature representing whether a physical defect exists;

[0014] Inputting the multi-level first target defect feature and the second target defect feature into the feature fusion structure to fuse them into a third target defect feature;

[0015] The third target defect feature is respectively input into a plurality of the classification head structures to classify the types of physical defects existing in the wafer substrate.

[0016] A second aspect of the present invention provides an automatic optical detection device, comprising:

[0017] An original overhead image data acquisition module is used to acquire original overhead image data of a wafer substrate under the illumination of a first light source; the angle formed between the illumination direction of the first light source and the wafer substrate is a right angle;

[0018] An illumination image data acquisition module is used to sequentially acquire multiple frames of side illumination image data of a wafer substrate under the illumination of multiple second light sources; the angles formed between the illumination directions of each of the second light sources and the wafer substrate are all acute angles;

[0019] A target overhead image data generating module, used for adjusting the brightness of the original overhead image data according to the multiple frames of the side image data to obtain the target overhead image data;

[0020] A defect classification network loading module, used to load a preset defect classification network; the defect classification network includes a first feature extraction structure, a second feature extraction structure, a feature fusion structure and a plurality of classification head structures;

[0021] A first target defect feature extraction module, used for inputting the target overhead image data into the first feature extraction structure to extract a multi-level first target defect feature representing whether there is a physical defect;

[0022] A second target defect feature extraction module, used for inputting multiple frames of the side illumination image data into the second feature extraction structure to jointly extract a second target defect feature representing whether there is a physical defect;

[0023] A third target defect feature fusion module, used for inputting the multi-level first target defect features and the second target defect features into the feature fusion structure to fuse them into a third target defect feature;

[0024] The physical defect classification module is used to input the third target defect feature into the multiple classification head structures respectively to classify the types of physical defects existing in the wafer substrate.

[0025] A third aspect of the present invention provides an automatic optical inspection device, the automatic optical inspection device comprising:

[0026] at least one processor; and

[0027] a memory communicatively connected to the at least one processor; wherein,

[0028] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the automatic optical detection method as described in the first aspect above.

[0029] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the automatic optical detection method as described in the first aspect above.

[0030] A fifth aspect of the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the automatic optical detection method as described in the first aspect above.

[0031] In this embodiment, original overhead image data is collected from a wafer substrate under the illumination of a first light source; the angle formed between the direction of illumination of the first light source and the wafer substrate is a right angle; multiple frames of side illumination image data are collected from the wafer substrate under the illumination of multiple second light sources in turn; the angles formed between the directions of illumination of each second light source and the wafer substrate are all acute angles; the brightness of the original overhead image data is adjusted according to the multiple frames of side illumination image data to obtain target overhead image data; a preset defect classification network is loaded; the defect classification network includes a first feature extraction structure, a second feature extraction structure, a feature fusion structure and multiple classification head structures; the target overhead image data is input into the first feature extraction structure to extract multi-level first target defect features representing whether there are physical defects; the multiple frames of side illumination image data are input into the second feature extraction structure to jointly extract second target defect features representing whether there are physical defects; the multi-level first target defect features and the second target defect features are input into the feature fusion structure to be fused into third target defect features; the third target defect features are respectively input into multiple classification head structures to classify the types of physical defects existing in the wafer substrate. This embodiment can form diversified image data by illuminating the wafer substrate with different light sources, adjust the brightness of the image data, and fuse the image data to extract common features, which can enrich the information content of the features, provide more samples in training the defect classification network, alleviate the overfitting of the defect classification network, and improve the performance of the defect classification network, thereby improving the accuracy of defect detection on the wafer substrate.

[0032] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 It is a flow chart of an automatic optical detection method provided in Embodiment 1 of the present invention.

[0035] Figure 2 It is a distribution diagram of a light source provided in Embodiment 1 of the present invention.

[0036] Figure 3 It is a structural diagram of a defect classification network provided in Example 1 of the present invention.

[0037] Figure 4It is a structural schematic diagram of an automatic optical detection device provided in Embodiment 2 of the present invention.

[0038] Figure 5 It is a structural schematic diagram of an automatic optical detection device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0040] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can cover sequential implementations other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0041] Embodiment 1

[0042] See also Figure 1 , shows a flow chart of an automatic optical detection method provided by the first embodiment of the present invention, the method can be performed by an automatic optical detection device, the automatic optical detection device can be implemented in the form of hardware and / or software, and the automatic optical detection device can be configured in an automatic optical detection device. Figure 1 As shown, the method includes:

[0043] Step 101: collecting original top-down image data of a wafer substrate under the illumination of a first light source.

[0044] The AOI equipment is equipped with an inspection table, a camera and multiple light sources. The wafer substrate to be inspected is loaded on the inspection table. Multiple light sources are distributed around the inspection table. The illumination range of each light source can cover the wafer substrate, and can provide lighting for the wafer substrate on the inspection table at different angles.

[0045] For the sake of distinction, the light source located in the vertical direction is recorded as the first light source, and the light source located in the acute angle direction is recorded as the second light source. Figure 2 As shown, the angle α formed between the irradiation direction of the first light source 201 and the plane where the wafer substrate 200 is located is a right angle (90°), and the angle β formed between the irradiation direction of each second light source 202 and the plane where the wafer substrate 200 is located is an acute angle (the value range is (0°, 90°), such as 37°).

[0046] Generally, the second light sources are evenly distributed around the wafer substrate, that is, multiple second light sources are located on the same horizontal plane, and the angle formed by the line between the irradiation directions of two adjacent second light sources and the midpoint of the wafer substrate is fixed.

[0047] For example, when three second light sources are distributed around the wafer substrate, the angle formed by the line connecting the irradiation directions of two adjacent second light sources and the midpoint of the wafer substrate is 120°.

[0048] When detecting whether there are physical defects on the wafer substrate in the AOI equipment, multiple second light sources can be turned off. At the same time, the first light source is started and driven to irradiate light to the wafer substrate. The camera is called to collect image data of the wafer substrate under its irradiation, which is recorded as the original overhead image data.

[0049] Step 102 : sequentially collect multiple frames of side illumination image data of the wafer substrate under the illumination of multiple second light sources.

[0050] When detecting whether there are physical defects on the wafer substrate in the AOI equipment, the first light source can be turned off. At the same time, multiple second light sources are started in sequence, and the second light sources are driven in sequence to irradiate light to the wafer substrate. Under the irradiation, the camera is called to collect multiple frames of image data of the wafer substrate in sequence, which are recorded as side-illumination image data.

[0051] When one of the second light sources is started, the other second light sources remain turned off. At this time, the currently started second light source is driven to irradiate light to the wafer substrate, and the camera is called to collect image data of the wafer substrate under the irradiation, which is recorded as side-illuminated image data.

[0052] When there are no physical defects in the wafer substrate, the original overhead image data and the side image data maintain a high degree of similarity in details such as brightness; when there are physical defects in the wafer substrate, due to differences in occlusion, reflection, etc. of the physical defects, there are certain differences in details such as brightness between the original overhead image data and the side image data.

[0053] In this embodiment, differentiated image data is generated by adding illumination from light sources at different angles, which can effectively increase the sample size for training the physical defect classification network, and help improve the performance of the physical defect classification network. In addition, multiple frames of image data are used as input at the same time, which can effectively increase the input information volume of the physical defect classification network, and help improve the accuracy of the physical defect classification network in detecting physical defects.

[0054] Step 103: adjusting the brightness of the original overhead image data according to the multiple frames of side-illumination image data to obtain target overhead image data.

[0055] In this embodiment, the multiple frames of side image data and the original overhead image data may be preprocessed, for example, by grayscale conversion, brightness adjustment, conversion to a specified size, etc., to improve the quality of the multiple frames of side image data and the original overhead image data.

[0056] For brightness adjustment, the brightness of the original overhead image data can be adjusted with reference to multiple frames of side image data to obtain target overhead image data. If there are physical defects in the wafer substrate, the difference in brightness between the original overhead image data and the side image data can be synchronized to the target overhead image data, thereby improving the correlation between the target overhead image data and the side image data, and increasing the amount of information in the target overhead image data, which is helpful for the physical defect classification network to detect physical defects.

[0057] In one embodiment of the present invention, step 103 may include the following steps:

[0058] Step 1031 : extract pixel points under the brightness channel from multiple frames of side illumination image data respectively to obtain multiple frames of original illumination image data.

[0059] For multiple frames of side-illuminated image data, pixel points under brightness channels (such as Y channel, L channel, etc.) can be extracted in their color space (such as YUV space, HSL space, etc.) to form new image data, which is recorded as original illumination image data.

[0060] Step 1032: merge multiple frames of original illumination image data into target illumination image data.

[0061] In this embodiment, multiple frames of original illumination image data may be combined into a new frame of image data in a linear or nonlinear manner, which is recorded as target illumination image data.

[0062] In a specific implementation, an algorithm such as OTSU (maximum between-class variance method) may be used to perform a binarization operation on each frame of original illumination image data to obtain candidate illumination image data.

[0063] The average brightness of each frame of candidate illumination image data is calculated to obtain an average brightness value.

[0064] The weights of the original illumination image data are configured according to the average brightness value using methods such as geometric scaling; wherein the weight is positively correlated with the average brightness value, that is, the larger the average brightness value, the larger the weight, and vice versa, the smaller the average brightness value, the smaller the weight.

[0065] The products of each pixel (brightness value) in multiple frames of original illumination image data and the corresponding weights are superimposed to obtain target illumination image data.

[0066] Step 1033: Under the condition that the original overhead image data is modeled as a fusion of brightness image data and reflection image data, the target illumination image data and the reflection image data are fused into target overhead image data.

[0067] In practical applications, the Retinex model believes that the color of an object is determined by the object's ability to reflect long-wave (red), medium-wave (green) and short-wave (blue) light, rather than by the absolute value of the reflected light intensity, and the color of the object is not affected by the non-uniformity of illumination and is consistent. That is, the Retinex model is based on color consistency (color constancy) by default. The so-called consistency refers to the ability of the human eye to recognize the original color of an object at different brightness levels.

[0068] Furthermore, in the Retinex model, incident light is irradiated, reflected by the object and enters the imaging system, forming the image seen by the user's glasses. In this process, the reflectivity is determined by the object itself and is not affected by the incident light, expressed as: L = I·T, where L is the image data received by the camera (such as the original overhead image data), I is the illumination component of the ambient light (such as brightness image data or target illumination image data), and T is the reflection component of the object carrying the image detail information (such as reflection image data).

[0069] In this embodiment, the target illumination image data determines the dynamic range that each pixel in the original overhead image data can reach, and the reflection image data determines the intrinsic properties of the original overhead image data.

[0070] The Retinex model is applied to separate the reflected image data from the original overhead image data, and the properties of the brightness image data are discarded from the original overhead image data L, thereby separating the original appearance of the object (such as a wafer substrate).

[0071] Thereafter, the target illumination image data and the reflected image data are synthesized to obtain the target overhead image data, thereby adding the brightness changes under different light sources to the object (such as a wafer substrate). The process is expressed as: L'=I'·T, where L' is the target overhead image data, I' is the target illumination image data, and T is the reflected image data.

[0072] Step 104: Load a preset defect classification network.

[0073] In this embodiment, a defect classification network can be constructed and trained in advance based on deep learning, where Figure 3 As shown, the defect classification network includes a first feature extraction structure Backbone_1, a second feature extraction structure Backbone_2, a feature fusion structure Neck and multiple classification head structures Head.

[0074] The first feature extraction structure Backbone_1 is used to extract multi-level features representing whether there are physical defects from image data of a wafer substrate. The so-called multi-level means that the scales of multiple features are not the same.

[0075] The second feature extraction structure Backbone_2 is used to jointly extract multi-level features that characterize whether there are physical defects from multiple frames of image data containing wafer substrates.

[0076] The feature fusion structure Neck is used to fuse and adjust features (such as dimensionality reduction, etc.) to better adapt to the classification task of physical defects in wafer substrate detection.

[0077] Multiple classification head structures Head are responsible for performing the classification task of detecting physical defects on wafer substrates, wherein each classification head structure Head corresponds to a type of physical defect, that is, each classification head structure Head is responsible for detecting whether the wafer substrate has one of the physical defects.

[0078] Generally speaking, the boundaries between certain physical defects in wafer substrates are relatively fuzzy, and there is uncertainty when labeling samples. For example, the boundary between damage and cracks is relatively fuzzy. For the same physical defect, some technicians may classify it as damage, while some technicians may classify it as cracks, and so on, based on their understanding of the labeling specifications. If an end-to-end multi-classification model is trained with samples labeled with different physical defects, the multi-classification model will overfit, resulting in a decrease in the accuracy of the multi-classification model in detecting physical defects in wafer substrates.

[0079] Since the boundary between the two categories of whether the wafer substrate has physical defects is relatively clear, the first feature extraction structure and the second feature extraction structure can be trained separately in this embodiment so that they have the ability to extract features representing whether there are physical defects from the image data containing the wafer substrate.

[0080] In this way, the parameters learned by the first feature extraction structure and the second feature extraction structure can better characterize the characteristics of physical defects, provide universal and high-precision features in shallow structures, and effectively alleviate the defects caused by blurred boundaries of physical defects and reduced detection accuracy.

[0081] When training the first feature extraction structure, a binary classification (whether there are physical defects in the wafer substrate) classification head (such as one or more fully connected layers) is added to the first feature extraction structure, and a binary classification network is formed with the first feature extraction structure and the classification head. The binary classification network is trained, and when the training of the binary classification network is completed, the first feature extraction structure is retained and the classification head is discarded.

[0082] When training the second feature extraction structure, a binary classification (whether there are physical defects in the wafer substrate) classification head (such as one or more fully connected layers) is added to the second feature extraction structure, and a binary classification network is formed with the second feature extraction structure and the classification head. The binary classification network is trained, and when the training of the binary classification network is completed, the second feature extraction structure is retained and the classification head is discarded.

[0083] When the training of the first feature extraction structure and the second feature extraction structure is completed, the defect classification network is trained. At this time, the first feature extraction structure and the second feature extraction structure are not updated, but the feature fusion structure and the multiple classification head structures are updated.

[0084] Step 105: Input the target overhead image data into a first feature extraction structure to extract a multi-level first target defect feature representing whether there is a physical defect.

[0085] In practical applications, such as Figure 3 As shown, the target overhead image data is input into the first feature extraction structure Backbone_1, and the first feature extraction structure Backbone_1 extracts multi-level features representing whether there are physical defects from the target overhead image data, which are recorded as first target defect features.

[0086] In one embodiment of the present invention, Figure 3 As shown, the first feature extraction structure Backbone_1 includes a first overhead convolution block DCnvBlock_1, a second overhead convolution block DCnvBlock_2 and a third overhead convolution block DCnvBlock_3.

[0087] Among them, the first overhead convolution block DCnvBlock_1, the second overhead convolution block DCnvBlock_2 and the third overhead convolution block DCnvBlock_3 are all structures that combine Self-Attention (self-attention mechanism) with CNN (Convolutional Neural Network). Self-Attention can overcome the local limitations of CNN, thereby using Self-Attention to utilize more relational information to enhance the function of CNN.

[0088] There are two forms of combining Self-Attention with CNN. One form is to use Self-Attention as a building block in the CNN network, and the other form is to use Self-Attention and CNN as complementary parts.

[0089] For example, the combination of Self-Attention and CNN includes the following two stages:

[0090] In the first stage, a 1×1 convolution is used to project the input features into query, key, and value.

[0091] The second stage includes the calculation of attention weights and the aggregation of value matrices, that is, the aggregation of local features.

[0092] Correspondingly, the multi-level first target defect features include a first overhead defect feature, a second overhead defect feature, and a third overhead defect feature.

[0093] In this embodiment, the target overhead image data is input into the first overhead convolution block DConvBlock_1, and a convolution operation is performed under a self-attention mechanism to obtain a first overhead defect feature that characterizes whether a physical defect exists.

[0094] The first overhead defect feature is input into the second overhead convolution block DConvBlock_2, and a convolution operation is performed under the self-attention mechanism to obtain a second overhead defect feature that characterizes whether there is a physical defect.

[0095] The second overhead defect feature is input into the third overhead convolution block DCnvBlock_3, and a convolution operation is performed under the self-attention mechanism to obtain the third overhead defect feature that characterizes whether there is a physical defect.

[0096] Performing convolution operations under the self-attention mechanism can focus attention on the changes in brightness in the target overhead image data, increase the weight of the brightness changes in the target overhead image data, expand the receptive field of the target overhead image data, and establish the global correlation between the first overhead defect feature, the second overhead defect feature and the third overhead defect feature, thereby improving the quality of the first overhead defect feature, the second overhead defect feature and the third overhead defect feature.

[0097] Step 106: Input the multiple frames of side illumination image data into a second feature extraction structure to jointly extract a second target defect feature that characterizes whether a physical defect exists.

[0098] In practical applications, such as Figure 3As shown, multiple frames of side illumination image data can be input into the second feature extraction structure Backbone_2. The second feature extraction structure Backbone_2 combines the multiple frames of side illumination image data to extract features that characterize whether physical defects exist, which are recorded as second target defect features.

[0099] In one embodiment of the present invention, Figure 3 As shown, the second feature extraction structure Backbone_2 includes multiple side-illumination branch convolution blocks SConvBlock_1 and side-illumination backbone convolution blocks SConvBlock_2.

[0100] Among them, multiple side-illumination branch convolution blocks SConvBlock_1 and side-illumination trunk convolution blocks SConvBlock_2 are structures related to extracting image features in CNN, such as convolution layers, pooling layers, normalization layers, activation layers, etc. They can reuse the local structure of extracting image features in third-party networks, such as VGG (Visual Geometry Group), Resnet (residual network), DarkNet, etc., and can also be customized structures, which is not limited in this embodiment.

[0101] In this embodiment, the side illumination branch convolution block SConvBlock_1 and the side illumination image data are in a one-to-one correspondence, so the correspondence between multiple frames of side illumination image data and multiple side illumination branch convolution blocks can be determined.

[0102] According to the corresponding relationship, the side illumination image data of each frame is input into the corresponding side illumination branch convolution block SConvBlock_1 to perform a convolution operation to obtain the side illumination branch defect features.

[0103] Use Concat, Add and other functions to merge multiple side-illumination branch defect features into side-illumination trunk defect features.

[0104] The side illumination trunk defect feature is input into the side illumination trunk convolution block SConvBlock_2 to perform a convolution operation to obtain a second target defect feature that characterizes whether there is a physical defect.

[0105] Step 107: Input the multi-level first target defect feature and the second target defect feature into a feature fusion structure to fuse them into a third target defect feature.

[0106] In practical applications, such as Figure 3 As shown, the multi-level first target defect features and the second target defect features can be simultaneously input into the feature fusion structure Neck, and the feature fusion structure Neck fuses the multi-level first target defect features and the second target defect features into a new feature, which is recorded as the third target defect feature.

[0107] In one embodiment of the present invention, the feature fusion structure Neck includes a first fusion convolution block FusionBlock_1, a second fusion convolution block FusionBlock_2 and a third fusion convolution block FusionBlock_3.

[0108] Among them, the first fusion convolution block FusionBlock_1, the second fusion convolution block FusionBlock_2 and the third fusion convolution block FusionBlock_3 are all structures related to the integration of image features in CNN, such as convolution layer, pooling layer, fully connected layer, downsampling layer, etc. They can reuse the local structure of extracting image features in a third-party network, such as YOLO, etc., or they can be customized structures, which is not limited in this embodiment.

[0109] In this embodiment, three downsampling operations DownSample are performed on the second target defect feature to obtain a first side-illuminated defect feature P1, a second side-illuminated defect feature P2 and a third side-illuminated defect feature P3, wherein the size of the first side-illuminated defect feature P1 is larger than the size of the second side-illuminated defect feature P2, and the size of the second side-illuminated defect feature P2 is larger than the size of the third side-illuminated defect feature P3.

[0110] Use functions such as Concat and Add to fuse the first top-view defect feature and the first side-view defect feature P1 into a first fused defect feature;

[0111] The first fusion defect feature is input into the first fusion convolution block FusionBlock_1 to perform a convolution operation to obtain a second fusion defect feature.

[0112] Use Concat, Add and other functions to merge the second fused defect feature, the second top-view defect feature and the second side-view defect feature P2 into the third fused defect feature.

[0113] The third fusion defect feature is input into the second fusion convolution block FusionBlock_2 to perform a convolution operation to obtain a fourth fusion defect feature.

[0114] Use Concat, Add and other functions to merge the fourth fused defect feature, the third top-view defect feature and the third side-view defect feature P3 into the fifth fused defect feature.

[0115] The fifth fusion defect feature is input into the third fusion convolution block FusionBlock_3 to perform a convolution operation to obtain the third target defect feature.

[0116] Step 108: Input the third target defect feature into a plurality of classification head structures respectively to classify the types of physical defects existing in the wafer substrate.

[0117] In practical applications, such as Figure 3 As shown, the third target defect feature is input into multiple classification head structures Head respectively, and the multiple classification head structures Head perform multiple classification operations on the wafer substrate to classify the types of physical defects existing in the wafer substrate.

[0118] In one embodiment of the present invention, Figure 3 As shown in the figure, each classification head structure includes a classification convolution block ClassConvBlock and an activation layer ActivationLayer; wherein the classification convolution block ClassConvBlock is a structure related to extracting image features in CNN, such as one or more convolution layers, etc., and the activation layer ActivationLayer includes a fully connected layer, an activation function (such as Softmax, etc.) and other structures.

[0119] In each classification head structure Head, the third target defect feature is input into the classification convolution block ClassConvBlock to perform a convolution operation to obtain the classification defect feature, and the classification defect feature is input into the activation layer ActivationLayer to be mapped into the probability of the existence of physical defects of the corresponding type in the wafer substrate.

[0120] The type of physical defects existing in the wafer substrate is determined according to the probability output by each classification head structure Head.

[0121] In a specific implementation, a classification header structure that meets a first condition is searched as a first target header structure; wherein the first condition is that the probability is greater than or equal to a preset first threshold.

[0122] If the first target head structure is found, it means that there is a physical defect in the wafer substrate and the boundary of the physical defect is relatively clear, then it is determined that there is a physical defect of the type corresponding to the first target head structure in the wafer substrate.

[0123] If the first target header structure is not found, at least two classification header structures meeting the second condition are searched as the second target header structure; wherein the second condition is that the probability is greater than or equal to a preset second threshold and less than a preset first threshold; and the first threshold is greater than the second threshold;

[0124] If the second target head structure is found, it means that there is a physical defect in the wafer substrate and the boundary of the physical defect is relatively fuzzy, then it is determined that there is a physical defect of a composite type corresponding to the second target head structure in the wafer substrate.

[0125] In this embodiment, original overhead image data is collected from a wafer substrate under the illumination of a first light source; the angle formed between the direction of illumination of the first light source and the wafer substrate is a right angle; multiple frames of side illumination image data are collected from the wafer substrate under the illumination of multiple second light sources in turn; the angles formed between the directions of illumination of each second light source and the wafer substrate are all acute angles; the brightness of the original overhead image data is adjusted according to the multiple frames of side illumination image data to obtain target overhead image data; a preset defect classification network is loaded; the defect classification network includes a first feature extraction structure, a second feature extraction structure, a feature fusion structure and multiple classification head structures; the target overhead image data is input into the first feature extraction structure to extract multi-level first target defect features representing whether there are physical defects; the multiple frames of side illumination image data are input into the second feature extraction structure to jointly extract second target defect features representing whether there are physical defects; the multi-level first target defect features and the second target defect features are input into the feature fusion structure to be fused into third target defect features; the third target defect features are respectively input into multiple classification head structures to classify the types of physical defects existing in the wafer substrate. This embodiment can form diversified image data by illuminating the wafer substrate with different light sources, adjust the brightness of the image data, and fuse the image data to extract common features, which can enrich the information content of the features, provide more samples in training the defect classification network, alleviate the overfitting of the defect classification network, and improve the performance of the defect classification network, thereby improving the accuracy of defect detection on the wafer substrate.

[0126] Embodiment 2

[0127] See also Figure 4 , shows a schematic diagram of the structure of an automatic optical detection device provided by Embodiment 2 of the present invention. Figure 4 As shown, the device comprises:

[0128] The original overhead image data acquisition module 401 is used to acquire original overhead image data of the wafer substrate under the illumination of a first light source; the angle formed between the illumination direction of the first light source and the wafer substrate is a right angle;

[0129] The side illumination image data acquisition module 402 is used to sequentially acquire multiple frames of side illumination image data of the wafer substrate under the illumination of multiple second light sources; the angles formed between the illumination directions of each of the second light sources and the wafer substrate are all acute angles;

[0130] A target overhead image data generating module 403 is used to adjust the brightness of the original overhead image data according to the multiple frames of the side image data to obtain target overhead image data;

[0131] The defect classification network loading module 404 is used to load a preset defect classification network; the defect classification network includes a first feature extraction structure, a second feature extraction structure, a feature fusion structure and a plurality of classification head structures;

[0132] The first target defect feature extraction module 405 is used to input the target overhead image data into the first feature extraction structure to extract a multi-level first target defect feature representing whether there is a physical defect;

[0133] A second target defect feature extraction module 406 is used to input multiple frames of the side illumination image data into the second feature extraction structure to jointly extract a second target defect feature representing whether there is a physical defect;

[0134] A third target defect feature fusion module 407 is used to input the multi-level first target defect features and the second target defect features into the feature fusion structure to fuse them into a third target defect feature;

[0135] The physical defect classification module 408 is used to input the third target defect feature into the multiple classification head structures to classify the types of physical defects existing in the wafer substrate.

[0136] In one embodiment of the present invention, the target overhead image data generating module 403 includes:

[0137] An original illumination image data extraction module is used to extract pixel points under the brightness channel from multiple frames of the side illumination image data to obtain multiple frames of original illumination image data;

[0138] A target illumination image data merging module, used for merging multiple frames of original illumination image data into target illumination image data;

[0139] The target overhead image data fusion module is used to fuse the target illumination image data and the reflection image data into target overhead image data under the condition that the original overhead image data is modeled as a fusion of brightness image data and reflection image data.

[0140] In one embodiment of the present invention, the target illumination image data merging module is further used for:

[0141] Performing a binarization operation on the original illumination image data of each frame to obtain candidate illumination image data;

[0142] Counting average brightness values ​​of the candidate illumination image data;

[0143] According to the average brightness value, weights are configured for the original illumination image data; the weights are positively correlated with the average brightness value;

[0144] The product of multiple frames of the original illumination image data and the weights is superimposed to obtain target illumination image data.

[0145] In one embodiment of the present invention, the first feature extraction structure includes a first overhead convolution block, a second overhead convolution block and a third overhead convolution block; the multi-level first target defect feature includes a first overhead defect feature, a second overhead defect feature and a third overhead defect feature;

[0146] The first target defect feature extraction module 405 is also used for:

[0147] Inputting the target overhead image data into the first overhead convolution block, performing a convolution operation under a self-attention mechanism, and obtaining a first overhead defect feature representing whether a physical defect exists;

[0148] Inputting the first overhead defect feature into the second overhead convolution block, performing a convolution operation under a self-attention mechanism, and obtaining a second overhead defect feature characterizing whether a physical defect exists;

[0149] The second overhead defect feature is input into the third overhead convolution block, and a convolution operation is performed under a self-attention mechanism to obtain a third overhead defect feature that characterizes whether there is a physical defect.

[0150] In one embodiment of the present invention, the second feature extraction structure includes a plurality of side-illumination branch convolution blocks and a side-illumination trunk convolution block;

[0151] The second target defect feature extraction module 406 is also used for:

[0152] Determine a correspondence between a plurality of frames of the side illumination image data and a plurality of the side illumination branch convolution blocks;

[0153] According to the corresponding relationship, the side illumination image data is input into the side illumination branch convolution block to perform a convolution operation to obtain a side illumination branch defect feature;

[0154] Merging the plurality of side-illumination branch defect features into a side-illumination trunk defect feature;

[0155] The side-illumination trunk defect feature is input into the side-illumination trunk convolution block to perform a convolution operation to obtain a second target defect feature that characterizes whether a physical defect exists.

[0156] In one embodiment of the present invention, the feature fusion structure includes a first fusion convolution block, a second fusion convolution block and a third fusion convolution block;

[0157] The third target defect feature fusion module 407 is also used for:

[0158] Performing three downsampling operations on the second target defect feature respectively to obtain a first side-illumination defect feature, a second side-illumination defect feature, and a third side-illumination defect feature;

[0159] Merging the first top-illumination defect feature and the first side-illumination defect feature into a first fused defect feature;

[0160] Inputting the first fused defect feature into the first fused convolution block to perform a convolution operation to obtain a second fused defect feature;

[0161] Merging the second fused defect feature, the second top-view defect feature and the second side-view defect feature into a third fused defect feature;

[0162] Inputting the third fusion defect feature into the second fusion convolution block to perform a convolution operation to obtain a fourth fusion defect feature;

[0163] Merging the fourth fused defect feature, the third top-view defect feature and the third side-view defect feature into a fifth fused defect feature;

[0164] The fifth fused defect feature is input into the third fused convolution block to perform a convolution operation to obtain a third target defect feature.

[0165] In one embodiment of the present invention, each of the classification head structures includes a classification convolution block and an activation layer; each of the classification head structures corresponds to a type of physical defect;

[0166] The physical defect classification module 408 includes:

[0167] A probability calculation module is used to input the third target defect feature into the classification convolution block in each of the classification head structures to perform a convolution operation to obtain a classification defect feature, and input the classification defect feature into the activation layer to map the probability that the wafer substrate has a physical defect of the type;

[0168] A probability determination module is used to determine the type of physical defects existing in the wafer substrate according to the probability.

[0169] In one embodiment of the present invention, the probability determination module is further used to:

[0170] Searching for the classification header structure that meets a first condition as a first target header structure; the first condition is that the probability is greater than or equal to a preset first threshold;

[0171] If found, it is determined that the wafer substrate has a physical defect of the type corresponding to the first target head structure;

[0172] If not found, searching for at least two of the classification head structures that meet the second condition as the second target head structure; the second condition is that the probability is greater than or equal to a preset second threshold and less than a preset first threshold; the first threshold is greater than the second threshold;

[0173] If found, it is determined that the wafer substrate has a physical defect corresponding to the second target head structure and of the composite type.

[0174] The automatic optical inspection device provided in the embodiment of the present invention can execute the automatic optical inspection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the automatic optical inspection method.

[0175] Embodiment 3

[0176] See also Figure 5 , shows a schematic diagram of the structure of an automatic optical inspection device provided by an embodiment of the present invention. The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0177] like Figure 5 As shown, the automatic optical inspection device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the automatic optical inspection device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0178] A number of components in the automated optical inspection device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the automated optical inspection device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0179] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as the automatic optical detection method.

[0180] In some embodiments, the automatic optical detection method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the automatic optical detection device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the automatic optical detection method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the automatic optical detection method in any other suitable manner (for example, by means of firmware).

[0181] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0182] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0183] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0184] To provide interaction with a user, the systems and techniques described herein can be implemented on an automated optical inspection device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or trackball), through which a user can provide input to the automated optical inspection device. Other types of devices can also be used to provide interaction with a user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0185] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0186] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0187] Embodiment 4

[0188] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the automatic optical detection method provided by any embodiment of the present invention is implemented.

[0189] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0190] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0191] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An automatic optical detection method, characterized in that: include: Collecting original top-down image data of a wafer substrate under the illumination of a first light source; the angle formed between the illumination direction of the first light source and the wafer substrate is a right angle; sequentially collecting multiple frames of side illumination image data of the wafer substrate under the illumination of multiple second light sources; the angles formed between the illumination directions of each of the second light sources and the wafer substrate are all acute angles; Adjusting the brightness of the original overhead image data according to the multiple frames of the side-illuminated image data to obtain target overhead image data; Load the preset defect classification network; The defect classification network includes a first feature extraction structure, a second feature extraction structure, a feature fusion structure and a plurality of classification head structures; Inputting the target overhead image data into the first feature extraction structure to extract a multi-level first target defect feature representing whether there is a physical defect; Inputting the plurality of frames of the side illumination image data into the second feature extraction structure to jointly extract a second target defect feature representing whether a physical defect exists; Inputting the multi-level first target defect feature and the second target defect feature into the feature fusion structure to fuse them into a third target defect feature; The third target defect feature is respectively input into a plurality of the classification head structures to classify the types of physical defects existing in the wafer substrate.

2. The method according to claim 1, characterized in that The step of adjusting the brightness of the original overhead image data according to the multiple frames of the side-illuminated image data to obtain the target overhead image data includes: Extracting pixel points under the brightness channel from multiple frames of the side illumination image data respectively to obtain multiple frames of original illumination image data; Merging multiple frames of original illumination image data into target illumination image data; Under the condition that the original overhead image data is modeled as a fusion of brightness image data and reflection image data, the target illumination image data and the reflection image data are fused into target overhead image data.

3. The method according to claim 2, characterized in that The step of merging the multiple frames of original illumination image data into target illumination image data comprises: Performing a binarization operation on the original illumination image data of each frame to obtain candidate illumination image data; Counting average brightness values ​​of the candidate illumination image data; According to the average brightness value, weights are configured for the original illumination image data; the weights are positively correlated with the average brightness value; The product of multiple frames of the original illumination image data and the weights is superimposed to obtain target illumination image data.

4. The method according to any one of claims 1 to 3, characterized in that The first feature extraction structure includes a first overhead convolution block, a second overhead convolution block and a third overhead convolution block; the multi-level first target defect feature includes a first overhead defect feature, a second overhead defect feature and a third overhead defect feature; The step of inputting the target overhead image data into the first feature extraction structure to extract a multi-level first target defect feature indicating whether there is a physical defect comprises: Inputting the target overhead image data into the first overhead convolution block, performing a convolution operation under a self-attention mechanism, and obtaining a first overhead defect feature representing whether a physical defect exists; Inputting the first overhead defect feature into the second overhead convolution block, performing a convolution operation under a self-attention mechanism, and obtaining a second overhead defect feature characterizing whether a physical defect exists; The second overhead defect feature is input into the third overhead convolution block, and a convolution operation is performed under a self-attention mechanism to obtain a third overhead defect feature that characterizes whether there is a physical defect.

5. The method according to claim 4, characterized in that The second feature extraction structure includes a plurality of side-illumination branch convolution blocks and a side-illumination trunk convolution block; The step of inputting the plurality of frames of the side illumination image data into the second feature extraction structure to jointly extract the second target defect feature representing whether a physical defect exists includes: Determine a correspondence between a plurality of frames of the side illumination image data and a plurality of the side illumination branch convolution blocks; According to the corresponding relationship, the side illumination image data is input into the side illumination branch convolution block to perform a convolution operation to obtain a side illumination branch defect feature; Merging the plurality of side-illumination branch defect features into a side-illumination trunk defect feature; The side-illumination trunk defect feature is input into the side-illumination trunk convolution block to perform a convolution operation to obtain a second target defect feature that characterizes whether a physical defect exists.

6. The method according to claim 5, characterized in that The feature fusion structure includes a first fusion convolution block, a second fusion convolution block and a third fusion convolution block; The step of inputting the multi-level first target defect features and the second target defect features into the feature fusion structure to fuse them into a third target defect feature comprises: Performing three downsampling operations on the second target defect feature respectively to obtain a first side-illumination defect feature, a second side-illumination defect feature, and a third side-illumination defect feature; Merging the first top-illumination defect feature and the first side-illumination defect feature into a first fused defect feature; Inputting the first fused defect feature into the first fused convolution block to perform a convolution operation to obtain a second fused defect feature; Merging the second fused defect feature, the second top-view defect feature and the second side-view defect feature into a third fused defect feature; Inputting the third fusion defect feature into the second fusion convolution block to perform a convolution operation to obtain a fourth fusion defect feature; Merging the fourth fused defect feature, the third top-view defect feature and the third side-view defect feature into a fifth fused defect feature; The fifth fused defect feature is input into the third fused convolution block to perform a convolution operation to obtain a third target defect feature.

7. The method according to claim 6, characterized in that Each of the classification head structures includes a classification convolution block and an activation layer; each of the classification head structures corresponds to a type of physical defect; Inputting the third target defect feature into the plurality of classification head structures to classify the types of physical defects existing in the wafer substrate comprises: In each of the classification head structures, the third target defect feature is input into the classification convolution block to perform a convolution operation to obtain a classification defect feature, and the classification defect feature is input into the activation layer to be mapped into the probability of the wafer substrate having a physical defect of the type; The type of physical defects present in the wafer substrate is determined according to the probability.

8. The method according to claim 7, characterized in that Determining the type of physical defects present in the wafer substrate according to the probability includes: Searching for the classification header structure that meets a first condition as a first target header structure; the first condition is that the probability is greater than or equal to a preset first threshold; If found, it is determined that the wafer substrate has a physical defect of the type corresponding to the first target head structure; If not found, searching for at least two of the classification head structures that meet the second condition as the second target head structure; the second condition is that the probability is greater than or equal to a preset second threshold and less than a preset first threshold; the first threshold is greater than the second threshold; If found, it is determined that the wafer substrate has a physical defect corresponding to the second target head structure and of the composite type.

9. An automatic optical inspection device, characterized in that: The automatic optical inspection equipment comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so as to enable the at least one processor to perform the automatic optical inspection method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the automatic optical detection method according to any one of claims 1 to 8 is implemented.

Images