An automatic optical detection method, device and storage medium

Abstract

The present invention discloses an automatic optical detection method, device and storage medium. The method includes: collecting original top-down image data of a wafer substrate under the illumination of a first light source; successively collecting multiple frames of side illumination image data of the wafer substrate under the illumination of multiple second light sources; adjusting the brightness of the original top-down image data according to the multiple frames of side illumination image data to obtain target top-down image data; loading a preset defect classification network; inputting the target top-down image data into a first feature extraction structure to extract multi-level first target defect features; inputting the multiple frames of side illumination image data into a second feature extraction structure to jointly extract second target defect features; inputting the multi-level first target defect features and the second target defect features into a feature fusion structure to fuse them into third target defect features; and inputting the third target defect features into multiple classification head structures respectively to classify the types of physical defects existing in the wafer substrate. The accuracy of detecting defects in the wafer substrate is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular, to an automated optical inspection method, device, and storage medium. Background Art

[0002] With the rapid development of high-power power supplies, motors, and other electronic products, electronic products are transforming towards miniaturization and high power density, posing higher requirements for the packaging of wafer substrates.

[0003] Wafer substrates are usually surface-mounted and packaged using processes such as TOLL (TO-Leadless). Before packaging, an automated optical inspection (AOI) device is used to detect whether there are physical defects on the wafer substrate, such as bumps, breakages, cracks, air holes, foreign objects, contamination, etc., to ensure product quality.

[0004] Currently, deep learning is gradually used in AOI devices to detect whether there are defects on wafer substrates. A large number of samples with physical defects are used to train a neural network to learn the ability to detect defects.

[0005] However, in the actual production process, the probability of defects occurring in wafer substrates is relatively low, resulting in a small number of samples with physical defects, leading to overfitting of the neural network, reducing the performance of the neural network, and lowering the accuracy of defect detection for wafer substrates. Summary of the Invention

[0006] In view of this, the present invention provides an automated optical inspection method, device, and storage medium to improve the accuracy of detecting physical defects on wafer substrates.

[0007] The first aspect of the present invention provides an automated optical inspection method, including:

[0008] Collecting original top-down image data of a wafer substrate under the irradiation of a first light source; the angle formed between the irradiation direction of the first light source and the wafer substrate is a right angle;

[0009] Successively collecting multiple frames of side illumination image data of the wafer substrate under the irradiation of multiple second light sources; the angle formed between the irradiation direction of each second light source and the wafer substrate is an acute angle;

[0010] Adjusting the brightness of the original top-down image data according to the multiple frames of side illumination image data to obtain target top-down image data;

[0011] Loading a pre-set defect classification network; the defect classification network includes a first feature extraction structure, a second feature extraction structure, a feature fusion structure, and multiple classification head structures;

[0012] Input the target downward illumination image data into the first feature extraction structure to extract multi-level first target defect features characterizing the presence of physical defects;

[0013] Input multiple frames of the side illumination image data into the second feature extraction structure to jointly extract second target defect features characterizing the presence of physical defects;

[0014] Input the multi-level first target defect features and the second target defect features into the feature fusion structure to fuse them into third target defect features;

[0015] Input the third target defect features into multiple classification head structures respectively to classify the types of physical defects existing in the wafer substrate.

[0016] The second aspect of the present invention provides an automatic optical inspection device, including:

[0017] An original downward illumination image data acquisition module, configured to acquire original downward illumination image data of a wafer substrate under the illumination of a first light source; the included angle formed between the illumination direction of the first light source and the wafer substrate is a right angle;

[0018] A side illumination image data acquisition module, configured to sequentially acquire multiple frames of side illumination image data of the wafer substrate under the illumination of multiple second light sources; the included angle formed between the illumination direction of each second light source and the wafer substrate is an acute angle;

[0019] A target downward illumination image data generation module, configured to adjust the brightness of the original downward illumination image data according to multiple frames of the side illumination image data to obtain target downward illumination image data;

[0020] A defect classification network loading module, configured to load a preset defect classification network; the defect classification network includes a first feature extraction structure, a second feature extraction structure, a feature fusion structure, and multiple classification head structures;

[0021] A first target defect feature extraction module, configured to input the target downward illumination image data into the first feature extraction structure to extract multi-level first target defect features characterizing the presence of physical defects;

[0022] A second target defect feature extraction module, configured to input multiple frames of the side illumination image data into the second feature extraction structure to jointly extract second target defect features characterizing the presence of physical defects;

[0023] A third target defect feature fusion module, configured to input the multi-level first target defect features and the second target defect features into the feature fusion structure to fuse them into third target defect features;

[0024] A physical defect classification module for inputting the third target defect features into a plurality of the classification head structures respectively to classify the types of physical defects existing in the wafer substrate.

[0025] A third aspect of the present invention provides an automatic optical detection device, the automatic optical detection device comprising:

[0026] At least one processor; and

[0027] A memory communicatively connected to the at least one processor; wherein,

[0028] The memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor is enabled to execute the automatic optical detection method as described in the first aspect above.

[0029] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the automatic optical detection method as described in the first aspect above is implemented.

[0030] A fifth aspect of the present invention provides a computer program product comprising a computer program, and when the computer program is executed by a processor, the automatic optical detection method as described in the first aspect above is implemented.

[0031] In this embodiment, under the illumination of a first light source, original downward illumination image data of a wafer substrate is collected; the included angle formed between the illumination direction of the first light source and the wafer substrate is a right angle; multiple frames of side illumination image data of the wafer substrate are collected in sequence under the illumination of multiple second light sources; the included angle formed between the illumination direction of each second light source and the wafer substrate is an acute angle; the brightness of the original downward illumination image data is adjusted according to the multiple frames of side illumination image data to obtain target downward illumination image data; a preset defect classification network is loaded; the defect classification network includes a first feature extraction structure, a second feature extraction structure, a feature fusion structure, and multiple classification head structures; the target downward illumination image data is input into the first feature extraction structure to extract multi-level first target defect features indicating the presence of physical defects; the multiple frames of side illumination image data are input into the second feature extraction structure to jointly extract second target defect features indicating the presence of physical defects; the multi-level first target defect features and the second target defect features are input into the feature fusion structure to be fused into third target defect features; the third target defect features are respectively input into the multiple classification head structures to classify the types of physical defects existing in the wafer substrate. In this embodiment, by illuminating the wafer substrate with different light sources, diverse image data can be formed, adjusting the brightness of the image data and fusing the image data to extract common features can enrich the information volume of the features, provide more samples for training the defect classification network, alleviate the overfitting situation of the defect classification network, improve the performance of the defect classification network, and thus improve the accuracy of detecting defects in the wafer substrate.

[0032] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. Description of the Drawings

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0034] Figure 1 is a flowchart of an automatic optical detection method provided in Embodiment 1 of the present invention.

[0035] Figure 2 is a schematic distribution diagram of a light source provided in Embodiment 1 of the present invention.

[0036] Figure 3 is a schematic structural diagram of a defect classification network provided in Embodiment 1 of the present invention.

[0037] Figure 4It is a schematic structural diagram of an automatic optical detection device provided in the second embodiment of the present invention.

[0038] Figure 5 It is a schematic structural diagram of an automatic optical detection device provided in the third embodiment of the present invention. Detailed implementation manners

[0039] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0040] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can cover the sequential implementations other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those clearly listed steps or units, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0041] Embodiment 1

[0042] Refer to Figure 1 , which shows a flowchart of an automatic optical detection method provided in Embodiment 1 of the present invention. This method can be executed by an automatic optical detection device, which can be implemented in the form of hardware and / or software, and this automatic optical detection device can be configured in an automatic optical detection device. As Figure 1 shown, this method includes:

[0043] Step 101: Collect original downward illumination image data for the wafer substrate under the illumination of the first light source.

[0044] In the AOI device, there are a detection table, a camera and multiple light sources. The wafer substrate to be detected is loaded on the detection table, and the multiple light sources are distributed around the detection table. The illumination range of each light source can cover the wafer substrate, and the wafer substrate on the detection table can be illuminated at different angles.

[0045] For the sake of distinction, the light source located in the vertical direction is denoted as the first light source, and the light source located in the acute angle direction is denoted as the second light source. That is, as Figure 2 shown, the included angle α formed between the irradiation direction of the first light source 201 and the plane where the wafer substrate 200 is located is a right angle (90°), and the included angle β formed between the irradiation direction of each second light source 202 and the plane where the wafer substrate 200 is located is an acute angle (the value range is (0°, 90°), such as 37°).

[0046] Generally, the second light sources are evenly distributed around the wafer substrate. That is, multiple second light sources are located on the same horizontal plane, and the included angle formed by the connection line between the irradiation directions of adjacent two second light sources and the midpoint of the wafer substrate is fixed.

[0047] For example, when three second light sources are distributed around the wafer substrate, the included angle formed by the connection line between the irradiation directions of adjacent two second light sources and the midpoint of the wafer substrate is 120° for each.

[0048] When detecting whether there are physical defects on the wafer substrate in the AOI device, multiple second light sources can be turned off. At the same time, the first light source is started, and the first light source is driven to irradiate light on the wafer substrate. The camera is called to collect image data of the wafer substrate under its irradiation, which is recorded as the original top-down image data.

[0049] Step 102: Sequentially collect multiple frames of side-view image data of the wafer substrate under the irradiation of multiple second light sources.

[0050] When detecting whether there are physical defects on the wafer substrate in the AOI device, the first light source can be turned off. At the same time, multiple second light sources are sequentially started in the order of multiple second light sources, and the second light sources are sequentially driven to irradiate light on the wafer substrate. The camera is called to sequentially collect multiple frames of image data of the wafer substrate under its irradiation, which is recorded as the side-view image data.

[0051] When starting one of the second light sources, the other second light sources remain off. At this time, the currently started second light source is driven to irradiate light on the wafer substrate, and the camera is called to collect image data of the wafer substrate under its irradiation, which is recorded as the side-view image data.

[0052] When there are no physical defects on the wafer substrate, the original top-down image data and the side-view image data maintain a high degree of similarity in details such as brightness; when there are physical defects on the wafer substrate, due to differences in occlusion, reflection, etc. of the physical defects, there are certain differences in details such as brightness between the original top-down image data and the side-view image data.

[0053] In this embodiment, by irradiating light sources at different angles to generate differential image data, the sample size for training the physical defect classification network can be effectively increased, which helps improve the performance of the physical defect classification network. Moreover, multiple frames of image data are used as inputs simultaneously, which can effectively increase the input information volume of the physical defect classification network and contribute to improving the accuracy of the physical defect classification network in detecting physical defects.

[0054] Step 103: Adjust the brightness of the original top-down image data according to multiple frames of side illumination image data to obtain target top-down image data.

[0055] In this embodiment, preprocessing can be performed on multiple frames of side illumination image data and the original top-down image data, such as grayscale conversion, brightness adjustment, conversion to a specified size, etc., to improve the quality of the multiple frames of side illumination image data and the original top-down image data.

[0056] For brightness adjustment, the brightness of the original top-down image data can be adjusted with reference to multiple frames of side illumination image data to obtain target top-down image data. If there are physical defects on the wafer substrate, the difference in brightness between the original top-down image data and the side illumination image data can be synchronized to the target top-down image data, improving the correlation between the target top-down image data and the side illumination image data and increasing the information volume of the target top-down image data, which helps the physical defect classification network detect physical defects.

[0057] In one embodiment of the present invention, step 103 may include the following steps:

[0058] Step 1031: Extract pixel points in the brightness channel from multiple frames of side illumination image data to obtain multiple frames of original illumination image data.

[0059] For multiple frames of side illumination image data, pixel points in the brightness channel (such as the Y channel, L channel, etc.) can be extracted in its color space (such as the YUV space, HSL space, etc.) to form new image data, denoted as the original illumination image data.

[0060] Step 1032: Merge multiple frames of original illumination image data into target illumination image data.

[0061] In this embodiment set, multiple frames of original illumination image data can be merged into a new frame of image data in a linear or non-linear manner, denoted as target illumination image data.

[0062] In specific implementation, algorithms such as OTSU (method of between-class variance) can be used to perform binarization operations on each frame of original illumination image data to obtain candidate illumination image data.

[0063] Statistical average brightness values for each frame of candidate illumination image data to obtain average brightness values.

[0064] Use methods such as equal ratio scaling to configure weights for the original illumination image data according to the average brightness value; among them, the weight is positively correlated with the average brightness value, that is, the larger the average brightness value, the larger the weight, and vice versa, the smaller the average brightness value, the smaller the weight.

[0065] Overlay the products of the pixel points (brightness values) and the corresponding weights in multiple frames of original illumination image data to obtain the target illumination image data.

[0066] Step 1033, under the condition that the original downward-looking image data is modeled as the fusion of the brightness image data and the reflection image data, fuse the target illumination image data and the reflection image data into the target downward-looking image data.

[0067] In practical applications, the Retinex model believes that the color of an object is determined by the object's reflection ability of long-wave (red), medium-wave (green), and short-wave (blue) light, rather than by the absolute value of the reflected light intensity, and the color of the object is not affected by the illumination non-uniformity and has consistency. That is, the Retinex model is default based on color sense consistency (color constancy). By consistency, it means the ability of the human eye to recognize the original color of an object under different brightnesses.

[0068] Furthermore, in the Retinex model, the incident light irradiates, is reflected by the object and enters the imaging system to form the picture seen by the user's glasses. In this process, the reflectivity is determined by the object itself and is not affected by the incident light, expressed as: L = I·T, where L is the image data received by the camera (such as the original downward-looking image data), I is the illumination component of the ambient light (such as the brightness image data or the target illumination image data), and T is the reflection component of the object carrying the picture details (such as the reflection image data).

[0069] In this embodiment, the target illumination image data determines the dynamic range that each pixel point in the original downward-looking image data can reach, and the reflection image data determines the inherent nature of the original downward-looking image data.

[0070] Apply the Retinex model to separate the reflection image data from the original downward-looking image data, and discard the nature of the brightness image data from the original downward-looking image data L, so as to separate the true appearance of the object (such as the wafer substrate).

[0071] After that, synthesize the target illumination image data and the reflection image data to obtain the target downward-looking image data, so as to add the changes in brightness under different light sources to the object (such as the wafer substrate). This process is expressed as: L' = I'·T, where L' is the target downward-looking image data, I' is the target illumination image data, and T is the reflection image data.

[0072] Step 104: Load the pre-set defect classification network.

[0073] In this embodiment, the defect classification network can be pre-constructed and trained based on deep learning. As shown, the defect classification network includes a first feature extraction structure Backbone_1, a second feature extraction structure Backbone_2, a feature fusion structure Neck, and multiple classification head structures Head. Figure 3 The first feature extraction structure Backbone_1 is used to extract multi-level features representing the presence or absence of physical defects from the image data of the wafer substrate. The so-called multi-level means that the scales of multiple features are different.

[0074] The second feature extraction structure Backbone_2 is used to jointly extract multi-level features representing the presence or absence of physical defects from multiple frames of image data of the wafer substrate.

[0075] The feature fusion structure Neck is used to perform processing such as fusing and adjusting (such as dimensionality reduction) on the features to better adapt to the classification task of detecting physical defects on the wafer substrate.

[0076] The multiple classification head structures Head are responsible for performing the classification task of detecting physical defects on the wafer substrate. Among them, each classification head structure Head corresponds to a type of physical defect, that is, each classification head structure Head is responsible for detecting whether a certain physical defect exists in the wafer substrate.

[0077] Generally, the boundaries between some physical defects in the wafer substrate are relatively blurred, and there is uncertainty when labeling samples. For example, the boundary between breakage and crack is relatively blurred. For the same physical defect, some technicians may classify it as breakage according to their understanding of the labeling specification, and some technicians may classify it as crack, etc. If a sample labeled with different physical defects is used to train an end-to-end multi-classification model, the multi-classification model will overfit, resulting in a decrease in the accuracy of the multi-classification model for detecting physical defects on the wafer substrate.

[0078] Since the boundary of the binary classification of whether there are physical defects in the wafer substrate is relatively clear, in this embodiment, the first feature extraction structure and the second feature extraction structure can be trained separately to enable them to have the ability to extract features representing the presence or absence of physical defects from the image data of the wafer substrate.

[0079] In this way, the parameters learned by the first feature extraction structure and the second feature extraction structure can better represent the features of physical defects, provide general and highly accurate features in the shallow structure, and effectively alleviate the defect that the blurred boundary of physical defects leads to a decrease in detection accuracy.

[0080] In this way, the parameters learned by the first feature extraction structure and the second feature extraction structure can better represent the features of physical defects, provide general and highly accurate features in the shallow structure, and effectively alleviate the defect that the blurred boundary of physical defects leads to a decrease in detection accuracy.

[0081] When training the first feature extraction structure, add a classification head (such as one or more fully connected layers) for binary classification (whether there are physical defects on the wafer substrate) to the first feature extraction structure, and form a binary classification network with the first feature extraction structure and the classification head. Train the binary classification network. When the training of the binary classification network is completed, retain the first feature extraction structure and discard the classification head.

[0082] When training the second feature extraction structure, add a classification head (such as one or more fully connected layers) for binary classification (whether there are physical defects on the wafer substrate) to the second feature extraction structure, and form a binary classification network with the second feature extraction structure and the classification head. Train the binary classification network. When the training of the binary classification network is completed, retain the second feature extraction structure and discard the classification head.

[0083] When the training of the first feature extraction structure and the second feature extraction structure is completed, train the defect classification network. At this time, do not update the first feature extraction structure and the second feature extraction structure, but update the feature fusion structure and multiple classification head structures.

[0084] Step 105: Input the target top-down image data into the first feature extraction structure to extract multi-level first target defect features representing whether there are physical defects.

[0085] In practical applications, as Figure 3 shown, input the target top-down image data into the first feature extraction structure Backbone_1. The first feature extraction structure Backbone_1 extracts multi-level features representing whether there are physical defects from the target top-down image data, denoted as the first target defect features.

[0086] In an embodiment of the present invention, as Figure 3 shown, the first feature extraction structure Backbone_1 includes a first top-down convolution block DConvBlock_1, a second top-down convolution block DConvBlock_2, and a third top-down convolution block DConvBlock_3.

[0087] Among them, the first top-down convolution block DConvBlock_1, the second top-down convolution block DConvBlock_2, and the third top-down convolution block DConvBlock_3 are all structures combining Self-Attention (self-attention mechanism) and CNN (Convolutional Neural Network). Self-Attention can overcome the limitations of the locality of CNN, so that Self-Attention can use more relational information to enhance the function of CNN.

[0088] The combination of Self-Attention and CNN includes two forms. One form is to use Self-Attention as a building block in the CNN network, and the other form is to use Self-Attention and CNN as complementary parts.

[0089] Exemplarily, the combination of Self-Attention and CNN includes the following two stages:

[0090] In the first stage, the input features are projected into query, key, and value using 1×1 convolution.

[0091] The second stage includes the calculation of attention weights and the aggregation of the value matrix, that is, aggregating local features.

[0092] Correspondingly, the multi-level first target defect features include the first front illumination defect feature, the second front illumination defect feature, and the third front illumination defect feature.

[0093] In this embodiment, the target front illumination image data is input into the first front illumination convolution block DConvBlock_1, and convolution operations are performed under the self-attention mechanism to obtain the first front illumination defect feature indicating whether there are physical defects.

[0094] The first front illumination defect feature is input into the second front illumination convolution block DConvBlock_2, and convolution operations are performed under the self-attention mechanism to obtain the second front illumination defect feature indicating whether there are physical defects.

[0095] The second front illumination defect feature is input into the third front illumination convolution block DConvBlock_3, and convolution operations are performed under the self-attention mechanism to obtain the third front illumination defect feature indicating whether there are physical defects.

[0096] Performing convolution operations under the self-attention mechanism can focus the attention on the brightness changes in the target front illumination image data, increase the weight of the brightness changes in the target front illumination image data, expand the receptive field of the target front illumination image data, establish the global correlation between the first front illumination defect feature, the second front illumination defect feature, and the third front illumination defect feature, thereby improving the quality of the first front illumination defect feature, the second front illumination defect feature, and the third front illumination defect feature.

[0097] Step 106: Input multiple frames of side illumination image data into the second feature extraction structure to jointly extract the second target defect feature indicating whether there are physical defects.

[0098] In practical applications, such as Figure 3As shown, multiple frames of side-view image data can be input into the second feature extraction structure Backbone_2. The second feature extraction structure Backbone_2 combines the multiple frames of side-view image data to extract features indicating whether there are physical defects, denoted as the second target defect features.

[0099] In one embodiment of the present invention, as Figure 3 shown, the second feature extraction structure Backbone_2 includes multiple side-view branch convolution blocks SConvBlock_1 and a side-view main convolution block SConvBlock_2.

[0100] Among them, the multiple side-view branch convolution blocks SConvBlock_1 and the side-view main convolution block SConvBlock_2 are both structures related to extracting image features in a CNN, such as convolutional layers, pooling layers, normalization layers, activation layers, etc. They can reuse local structures for extracting image features in third-party networks, such as VGG (Visual Geometry Group), Resnet (Residual Network), DarkNet, etc., or can also be custom structures, and this embodiment does not limit this.

[0101] In this embodiment, there is a one-to-one correspondence between the side-view branch convolution blocks SConvBlock_1 and the side-view image data. Then, the correspondence between the multiple frames of side-view image data and the multiple side-view branch convolution blocks can be determined.

[0102] According to the correspondence, each frame of side-view image data is input into the corresponding side-view branch convolution block SConvBlock_1 to perform a convolution operation, obtaining side-view branch defect features.

[0103] Functions such as Concat and Add are used to fuse the multiple side-view branch defect features into a side-view main defect feature.

[0104] The side-view main defect feature is input into the side-view main convolution block SConvBlock_2 to perform a convolution operation, obtaining the second target defect features indicating whether there are physical defects.

[0105] Step 107: Input the multi-level first target defect features and the second target defect features into a feature fusion structure to fuse them into third target defect features.

[0106] In practical applications, as Figure 3 shown, the multi-level first target defect features and the second target defect features can be simultaneously input into the feature fusion structure Neck. The feature fusion structure Neck fuses the multi-level first target defect features and the second target defect features into a new feature, denoted as the third target defect features.

[0107] In an embodiment of the present invention, the feature fusion structure Neck includes a first fusion convolution block FusionBlock_1, a second fusion convolution block FusionBlock_2, and a third fusion convolution block FusionBlock_3.

[0108] Among them, the first fusion convolution block FusionBlock_1, the second fusion convolution block FusionBlock_2, and the third fusion convolution block FusionBlock_3 are all structures related to integrating image features in a CNN, such as a convolutional layer, a pooling layer, a fully connected layer, a downsampling layer, etc. They can reuse local structures for extracting image features in a third-party network, such as YOLO, or can be custom structures. This embodiment does not limit this.

[0109] In this embodiment, three downsampling operations DownSample are respectively performed on the second target defect feature to obtain a first side illumination defect feature P1, a second side illumination defect feature P2, and a third side illumination defect feature P3. Among them, the size of the first side illumination defect feature P1 is larger than the size of the second side illumination defect feature P2, and the size of the second side illumination defect feature P2 is larger than the size of the third side illumination defect feature P3.

[0110] Use functions such as Concat and Add to fuse the first top illumination defect feature and the first side illumination defect feature P1 into a first fusion defect feature;

[0111] Input the first fusion defect feature into the first fusion convolution block FusionBlock_1 to perform a convolution operation to obtain a second fusion defect feature.

[0112] Use functions such as Concat and Add to fuse the second fusion defect feature, the second top illumination defect feature, and the second side illumination defect feature P2 into a third fusion defect feature.

[0113] Input the third fusion defect feature into the second fusion convolution block FusionBlock_2 to perform a convolution operation to obtain a fourth fusion defect feature.

[0114] Use functions such as Concat and Add to fuse the fourth fusion defect feature, the third top illumination defect feature, and the third side illumination defect feature P3 into a fifth fusion defect feature.

[0115] Input the fifth fusion defect feature into the third fusion convolution block FusionBlock_3 to perform a convolution operation to obtain a third target defect feature.

[0116] Step 108: Input the third target defect feature into multiple classification head structures respectively to classify the types of physical defects existing in the wafer substrate.

[0117] In practical applications, such asFigure 3 As shown, the third target defect feature is respectively input into multiple classification head structures Head, and the multiple classification head structures Head perform multi-classification operations on the wafer substrate to classify the types of physical defects existing in the wafer substrate.

[0118] In an embodiment of the present invention, as Figure 3 shown, each classification head structure includes a classification convolution block ClassConvBlock and an activation layer ActivationLayer; among them, the classification convolution block ClassConvBlock is a structure related to extracting image features in the CNN, such as one or more convolutional layers, etc., and the activation layer ActivationLayer includes structures such as a fully connected layer and an activation function (such as Softmax, etc.).

[0119] In each classification head structure Head, the third target defect feature is input into the classification convolution block ClassConvBlock to perform a convolution operation to obtain a classified defect feature, and the classified defect feature is input into the activation layer ActivationLayer to be mapped to the probability of the existence of a physical defect of the corresponding type in the wafer substrate.

[0120] Determine the type of physical defect existing in the wafer substrate according to the probability output by each classification head structure Head.

[0121] In a specific implementation, find a classification head structure that meets the first condition as the first target head structure; where the first condition is that the probability is greater than or equal to a preset first threshold.

[0122] If the first target head structure is found, it means that the wafer substrate has a physical defect and the boundary of the physical defect is relatively clear, then it is determined that the wafer substrate has a physical defect corresponding to the first target head structure.

[0123] If the first target head structure is not found, then find at least two classification head structures that meet the second condition as the second target head structures; where the second condition is that the probability is greater than or equal to a preset second threshold and less than a preset first threshold; the first threshold is greater than the second threshold;

[0124] If the second target head structure is found, it means that the wafer substrate has a physical defect and the boundary of the physical defect is relatively blurred, then it is determined that the wafer substrate has a physical defect of a composite type corresponding to the second target head structure.

[0125] In this embodiment, under the illumination of a first light source, original top-down image data of a wafer substrate is collected; the included angle formed between the illumination direction of the first light source and the wafer substrate is a right angle; multiple frames of side illumination image data of the wafer substrate are collected in sequence under the illumination of a plurality of second light sources; the included angle formed between the illumination direction of each second light source and the wafer substrate is an acute angle; the brightness of the original top-down image data is adjusted according to the multiple frames of side illumination image data to obtain target top-down image data; a preset defect classification network is loaded; the defect classification network includes a first feature extraction structure, a second feature extraction structure, a feature fusion structure, and a plurality of classification head structures; the target top-down image data is input into the first feature extraction structure to extract multi-level first target defect features characterizing the presence of physical defects; the multiple frames of side illumination image data are input into the second feature extraction structure to jointly extract second target defect features characterizing the presence of physical defects; the multi-level first target defect features and the second target defect features are input into the feature fusion structure to be fused into third target defect features; the third target defect features are respectively input into the plurality of classification head structures to classify the types of physical defects existing in the wafer substrate. In this embodiment, by illuminating the wafer substrate with different light sources, diverse image data can be formed. Adjusting the brightness of the image data and fusing the image data to extract common features can enrich the information content of the features, provide more samples for training the defect classification network, alleviate the overfitting of the defect classification network, improve the performance of the defect classification network, and thus improve the accuracy of detecting defects in the wafer substrate.

[0126] Embodiment 2

[0127] See Figure 4 , which shows a structural schematic diagram of an automatic optical detection device provided in Embodiment 2 of the present invention. As Figure 4 shown, the device includes:

[0128] An original top-down image data acquisition module 401, configured to collect original top-down image data of a wafer substrate under the illumination of a first light source; the included angle formed between the illumination direction of the first light source and the wafer substrate is a right angle;

[0129] A side illumination image data acquisition module 402, configured to collect multiple frames of side illumination image data of a wafer substrate in sequence under the illumination of a plurality of second light sources; the included angle formed between the illumination direction of each second light source and the wafer substrate is an acute angle;

[0130] A target top-down image data generation module 403, configured to adjust the brightness of the original top-down image data according to multiple frames of the side illumination image data to obtain target top-down image data;

[0131] A defect classification network loading module 404, configured to load a preset defect classification network; the defect classification network includes a first feature extraction structure, a second feature extraction structure, a feature fusion structure, and a plurality of classification head structures;

[0132] The first target defect feature extraction module 405 is configured to input the target top-down image data into the first feature extraction structure to extract multi-level first target defect features characterizing the presence or absence of physical defects;

[0133] The second target defect feature extraction module 406 is configured to input multiple frames of the side illumination image data into the second feature extraction structure to jointly extract second target defect features characterizing the presence or absence of physical defects;

[0134] The third target defect feature fusion module 407 is configured to input the multi-level first target defect features and the second target defect features into the feature fusion structure to fuse them into third target defect features;

[0135] The physical defect classification module 408 is configured to input the third target defect features into multiple classification head structures respectively to classify the types of physical defects existing in the wafer substrate.

[0136] In an embodiment of the present invention, the target top-down image data generation module 403 includes:

[0137] The original illumination image data extraction module is configured to extract pixel points in the luminance channel from multiple frames of the side illumination image data respectively to obtain multiple frames of original illumination image data;

[0138] The target illumination image data merging module is configured to merge multiple frames of the original illumination image data into target illumination image data;

[0139] The target top-down image data fusion module is configured to fuse the target illumination image data and the reflection image data into target top-down image data on the condition that the original top-down image data is modeled as a fusion of luminance image data and reflection image data.

[0140] In an embodiment of the present invention, the target illumination image data merging module is further configured to:

[0141] Perform a binarization operation on each frame of the original illumination image data to obtain candidate illumination image data;

[0142] Statistically calculate the average luminance value of the candidate illumination image data;

[0143] Configure weights for the original illumination image data according to the average luminance value; the weights are positively correlated with the average luminance value;

[0144] Superimpose the products of multiple frames of the original illumination image data and the weights to obtain target illumination image data.

[0145] In one embodiment of the present invention, the first feature extraction structure includes a first downward-facing convolutional block, a second downward-facing convolutional block, and a third downward-facing convolutional block; the multi-level first target defect features include a first downward-facing defect feature, a second downward-facing defect feature, and a third downward-facing defect feature;

[0146] The first target defect feature extraction module 405 is further configured to:

[0147] Input the target downward-facing image data into the first downward-facing convolutional block, and perform a convolutional operation under the self-attention mechanism to obtain a first downward-facing defect feature indicating whether there is a physical defect;

[0148] Input the first downward-facing defect feature into the second downward-facing convolutional block, and perform a convolutional operation under the self-attention mechanism to obtain a second downward-facing defect feature indicating whether there is a physical defect;

[0149] Input the second downward-facing defect feature into the third downward-facing convolutional block, and perform a convolutional operation under the self-attention mechanism to obtain a third downward-facing defect feature indicating whether there is a physical defect.

[0150] In one embodiment of the present invention, the second feature extraction structure includes a plurality of side-facing branch convolutional blocks and a side-facing main convolutional block;

[0151] The second target defect feature extraction module 406 is further configured to:

[0152] Determine the correspondence between multiple frames of the side-facing image data and the plurality of side-facing branch convolutional blocks;

[0153] Input the side-facing image data into the side-facing branch convolutional blocks according to the correspondence to perform a convolutional operation to obtain side-facing branch defect features;

[0154] Fuse the plurality of side-facing branch defect features into a side-facing main defect feature;

[0155] Input the side-facing main defect feature into the side-facing main convolutional block to perform a convolutional operation to obtain a second target defect feature indicating whether there is a physical defect.

[0156] In one embodiment of the present invention, the feature fusion structure includes a first fusion convolutional block, a second fusion convolutional block, and a third fusion convolutional block;

[0157] The third target defect feature fusion module 407 is further configured to:

[0158] Perform three downsampling operations on the second target defect feature respectively to obtain a first side-facing defect feature, a second side-facing defect feature, and a third side-facing defect feature;

[0159] Fuse the first top-down defect feature and the first side-down defect feature into a first fused defect feature;

[0160] Input the first fused defect feature into the first fused convolutional block to perform a convolutional operation to obtain a second fused defect feature;

[0161] Fuse the second fused defect feature, the second top-down defect feature and the second side-down defect feature into a third fused defect feature;

[0162] Input the third fused defect feature into the second fused convolutional block to perform a convolutional operation to obtain a fourth fused defect feature;

[0163] Fuse the fourth fused defect feature, the third top-down defect feature and the third side-down defect feature into a fifth fused defect feature;

[0164] Input the fifth fused defect feature into the third fused convolutional block to perform a convolutional operation to obtain a third target defect feature.

[0165] In an embodiment of the present invention, each of the classification head structures includes a classification convolutional block and an activation layer; each of the classification head structures corresponds to a type of physical defect;

[0166] The physical defect classification module 408 includes:

[0167] A probability calculation module, which is used to input the third target defect feature into the classification convolutional block in each of the classification head structures to perform a convolutional operation to obtain a classified defect feature, and input the classified defect feature into the activation layer to map it to the probability that the wafer substrate has a physical defect of the type;

[0168] A probability determination module, which is used to determine the type of physical defect existing in the wafer substrate according to the probability.

[0169] In an embodiment of the present invention, the probability determination module is further used for:

[0170] Search for the classification head structure that meets the first condition as the first target head structure; the first condition is that the probability is greater than or equal to a preset first threshold;

[0171] If found, it is determined that the wafer substrate has a physical defect corresponding to the first target head structure and of the type;

[0172] If not found, search for at least two of the classification head structures that meet the second condition as the second target head structures; the second condition is that the probability is greater than or equal to a preset second threshold and less than a preset first threshold; the first threshold is greater than the second threshold;

[0173] If found, it is determined that the wafer substrate has a physical defect corresponding to the second target head structure and combined with the type.

[0174] The automatic optical detection device provided by the embodiment of the present invention can execute the automatic optical detection method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the automatic optical detection method.

[0175] Embodiment III

[0176] Refer to Figure 5 , which shows a schematic structural diagram of an automatic optical detection device provided by an embodiment of the present invention. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0177] As Figure 5 shown, the automatic optical detection device 10 includes at least one processor 11 and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. Among them, the memory stores a computer program executable by the at least one processor. The processor 11 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the automatic optical detection device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0178] Multiple components in the automatic optical detection device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the automatic optical detection device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0179] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the automatic optical inspection method.

[0180] In some embodiments, the automatic optical inspection method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the automatic optical inspection device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the automatic optical inspection method described above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute the automatic optical inspection method by any other suitable means (e.g., by means of firmware).

[0181] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGA), application-specific integrated circuits (ASIC), application-specific standard products (ASSP), systems-on-chip (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor, and can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0182] The computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processors of a general-purpose computer, a special-purpose computer, or other programmable data processing devices, such that when the computer programs are executed by the processors, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer programs can be executed entirely on the machine, partially on the machine, as an independent software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0183] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0184] In order to provide interaction with a user, the systems and techniques described herein can be implemented on an automated optical inspection device that has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the automated optical inspection device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, speech input, or tactile input).

[0185] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0186] A computing system may include a client and a server. The client and the server are generally far from each other and usually interact via a communication network. The relationship between the client and the server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, solving the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0187] Embodiment 4

[0188] The embodiment of the present invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the automatic optical detection method provided in any embodiment of the present invention.

[0189] In the process of implementing the computer program product, computer program code for performing the operations of the present invention can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN) - or can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0190] It should be understood that various forms of the processes shown above can be used, reordering, adding or deleting steps. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is made herein.

[0191] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. An automatic optical detection method, characterized in that: include: Collecting original top-down image data of a wafer substrate under the illumination of a first light source; the angle formed between the illumination direction of the first light source and the wafer substrate is a right angle; sequentially collecting multiple frames of side illumination image data of the wafer substrate under the illumination of multiple second light sources; the angles formed between the illumination directions of each of the second light sources and the wafer substrate are all acute angles; Adjusting the brightness of the original overhead image data according to the multiple frames of the side-illuminated image data to obtain target overhead image data; Load the preset defect classification network; The defect classification network includes a first feature extraction structure, a second feature extraction structure, a feature fusion structure and a plurality of classification head structures; Inputting the target overhead image data into the first feature extraction structure to extract a multi-level first target defect feature representing whether there is a physical defect; Inputting the plurality of frames of the side illumination image data into the second feature extraction structure to jointly extract a second target defect feature representing whether a physical defect exists; Inputting the multi-level first target defect feature and the second target defect feature into the feature fusion structure to fuse them into a third target defect feature; Inputting the third target defect feature into the plurality of classification head structures respectively to classify the types of physical defects existing in the wafer substrate; Wherein, the first feature extraction structure includes a first overhead convolution block, a second overhead convolution block and a third overhead convolution block; the multi-level first target defect feature includes a first overhead defect feature, a second overhead defect feature and a third overhead defect feature; The step of inputting the target overhead image data into the first feature extraction structure to extract a multi-level first target defect feature indicating whether there is a physical defect comprises: Inputting the target overhead image data into the first overhead convolution block, performing a convolution operation under a self-attention mechanism, and obtaining a first overhead defect feature representing whether a physical defect exists; Inputting the first overhead defect feature into the second overhead convolution block, performing a convolution operation under a self-attention mechanism, and obtaining a second overhead defect feature characterizing whether a physical defect exists; Inputting the second overhead defect feature into the third overhead convolution block, performing a convolution operation under a self-attention mechanism, and obtaining a third overhead defect feature representing whether a physical defect exists; The second feature extraction structure includes a plurality of side-illumination branch convolution blocks and a side-illumination trunk convolution block; The step of inputting the plurality of frames of the side illumination image data into the second feature extraction structure to jointly extract the second target defect feature representing whether a physical defect exists includes: Determine a correspondence between a plurality of frames of the side illumination image data and a plurality of the side illumination branch convolution blocks; According to the corresponding relationship, the side illumination image data is input into the side illumination branch convolution block to perform a convolution operation to obtain a side illumination branch defect feature; Merging the plurality of side-illumination branch defect features into a side-illumination trunk defect feature; The side-illumination trunk defect feature is input into the side-illumination trunk convolution block to perform a convolution operation to obtain a second target defect feature that characterizes whether a physical defect exists.

2. The method according to claim 1, characterized in that The step of adjusting the brightness of the original overhead image data according to the multiple frames of the side-illuminated image data to obtain the target overhead image data includes: Extracting pixel points under the brightness channel from multiple frames of the side illumination image data respectively to obtain multiple frames of original illumination image data; Merging multiple frames of original illumination image data into target illumination image data; Under the condition that the original overhead image data is modeled as a fusion of brightness image data and reflection image data, the target illumination image data and the reflection image data are fused into target overhead image data.

3. The method according to claim 2, characterized in that The step of merging the multiple frames of original illumination image data into target illumination image data comprises: Performing a binarization operation on the original illumination image data of each frame to obtain candidate illumination image data; Counting average brightness values ​​of the candidate illumination image data; According to the average brightness value, weights are configured for the original illumination image data; the weights are positively correlated with the average brightness value; The product of multiple frames of the original illumination image data and the weights is superimposed to obtain target illumination image data.

4. The method according to claim 1, characterized in that: The feature fusion structure includes a first fusion convolution block, a second fusion convolution block and a third fusion convolution block; The step of inputting the multi-level first target defect features and the second target defect features into the feature fusion structure to fuse them into a third target defect feature comprises: Performing three downsampling operations on the second target defect feature respectively to obtain a first side-illumination defect feature, a second side-illumination defect feature, and a third side-illumination defect feature; Merging the first top-illumination defect feature and the first side-illumination defect feature into a first fused defect feature; Inputting the first fused defect feature into the first fused convolution block to perform a convolution operation to obtain a second fused defect feature; Merging the second fused defect feature, the second top-view defect feature and the second side-view defect feature into a third fused defect feature; Inputting the third fusion defect feature into the second fusion convolution block to perform a convolution operation to obtain a fourth fusion defect feature; Merging the fourth fused defect feature, the third top-view defect feature and the third side-view defect feature into a fifth fused defect feature; The fifth fused defect feature is input into the third fused convolution block to perform a convolution operation to obtain a third target defect feature.

5. The method according to claim 4, characterized in that Each of the classification head structures includes a classification convolution block and an activation layer; each of the classification head structures corresponds to a type of physical defect; Inputting the third target defect feature into the plurality of classification head structures to classify the types of physical defects existing in the wafer substrate comprises: In each of the classification head structures, the third target defect feature is input into the classification convolution block to perform a convolution operation to obtain a classification defect feature, and the classification defect feature is input into the activation layer to be mapped into the probability of the wafer substrate having a physical defect of the type; The type of physical defects present in the wafer substrate is determined according to the probability.

6. The method according to claim 5, characterized in that Determining the type of physical defects present in the wafer substrate according to the probability includes: Searching for the classification header structure that meets a first condition as a first target header structure; the first condition is that the probability is greater than or equal to a preset first threshold; If found, it is determined that the wafer substrate has a physical defect of the type corresponding to the first target head structure; If not found, searching for at least two of the classification head structures that meet the second condition as the second target head structure; the second condition is that the probability is greater than or equal to a preset second threshold and less than a preset first threshold; the first threshold is greater than the second threshold; If found, it is determined that the wafer substrate has a physical defect corresponding to the second target head structure and of the composite type.

7. An automatic optical inspection device, characterized in that: The automatic optical inspection equipment comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so as to enable the at least one processor to perform the automatic optical inspection method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the automatic optical detection method according to any one of claims 1 to 6 is implemented.

Images