Defect detection method and model training method, device, equipment and medium
Patent Information
- Application Number
- CN202611137106.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-29
- Publication Date
- 2026-10-27
AI Technical Summary
[0031]本公开提供的缺陷检测方法、模型训练方法、装置、设备及介质，通过获取待检测晶圆的第一图像，以及无缺陷晶圆的第二图像，并对第一图像和第二图像进行编码处理，得到第一图像对应的第一编码特征和第二图像对应的第二编码特征，对第一编码特征和第二编码特征进行信息融合，得到第三编码特征，实现了第一图像和第二图像的特征交互，通过特征交互得到的第三编码特征融合了第一图像和第二图像的信息，有利于准确地捕捉第一图像和第二图像之间的差异；通过对第一图像进行多尺度特征提取，基于第一图像的多尺度特征和第三编码特征，生成第一图像的特征图，基于第一图像的特征图进行缺陷识别，有利于提高缺陷识别对局部纹理和缺陷边界的敏感性，提高缺陷识别的准确性。
Smart Images
Figure CN122989602A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of semiconductor technology, and in particular to a defect detection method, model training method, apparatus, device, and medium. Background Technology
[0002] Wafers are the core substrate for chip manufacturing, and defect detection is crucial throughout the entire process, including photolithography, etching, and thin film deposition, making it a key step in controlling chip yield. Its core function is to promptly identify various microscopic defects such as particles, scratches, short circuits, and pattern misalignments. Accurate defect detection improves wafer yield, ensures production capacity, and has irreplaceable practical significance for improving the quality and efficiency of the semiconductor industry and ensuring supply chain stability. Therefore, improving the accuracy of wafer defect detection is a technical problem that needs to be solved. Summary of the Invention
[0003] Therefore, it is necessary to provide a defect detection method, model training method, device, equipment, and medium that can improve the accuracy of wafer defect detection in response to the above-mentioned technical problems.
[0004] In a first aspect, this disclosure provides a defect detection method, the method comprising: acquiring a first image of a wafer to be inspected and a second image of a defect-free wafer; performing multi-scale feature extraction on the first image to obtain multi-scale features of the first image; performing encoding processing on the first image and the second image respectively to obtain a first encoded feature corresponding to the first image and a second encoded feature corresponding to the second image; performing information fusion on the first encoded feature and the second encoded feature to obtain a third encoded feature; generating a feature map of the first image based on the third encoded feature and the multi-scale features; and identifying defects contained in the first image based on the feature map.
[0005] In some implementations, generating a feature map of the first image based on the third coding feature and the multi-scale feature includes: concatenating the third coding feature and the first coding feature to obtain a concatenated feature; inputting the concatenated feature and the multi-scale feature into a decoder, and generating a feature map of the first image through the decoder.
[0006] In some embodiments, the first coding feature includes a first primitive sequence of the first image, and the second coding feature includes a second primitive sequence of the second image; the step of fusing the first coding feature and the second coding feature to obtain a third coding feature includes: fusing the first primitive sequence and the second primitive sequence to obtain a third primitive sequence, wherein the third coding feature includes the third primitive sequence.
[0007] In some implementations, fusing information between the first primitive sequence and the second primitive sequence to obtain a third primitive sequence includes: performing cross-attention calculation on the first primitive sequence and the second primitive sequence to obtain the third primitive sequence.
[0008] In some implementations, the step of performing cross-attention calculation on the first primitive sequence and the second primitive sequence to obtain the third primitive sequence includes: performing bidirectional cross-attention calculation on the first primitive sequence and the second primitive sequence to obtain a calculation result, the calculation result including a primitive sequence obtained by forward cross-attention calculation and a primitive sequence obtained by reverse cross-attention calculation; and fusing the primitive sequences obtained by forward cross-attention calculation and the primitive sequences obtained by reverse cross-attention calculation to obtain the third primitive sequence.
[0009] Secondly, this disclosure provides a model training method, which includes: acquiring a first image of a wafer to be inspected, a second image of a defect-free wafer, and defect annotation information of the first image; inputting the first image and the second image into a preset model, and executing the method described in any embodiment of the first aspect through the preset model to obtain defects contained in the first image; determining a loss value of the preset model based on the defects in the first image and the defect annotation information of the first image; and adjusting the parameters of the preset model based on the loss value until the preset model converges to obtain a target model.
[0010] In some embodiments, before inputting the first image and the second image into a preset model, the method further includes at least one of the following operations: changing the brightness, contrast, or noise perturbation of the first image and the second image respectively; performing translation processing on the grain image contained in the first image; and fusing non-uniform illumination information in the first image.
[0011] Thirdly, this disclosure provides a defect detection device, comprising:
[0012] The acquisition module is used to acquire a first image of the wafer to be inspected and a second image of the defect-free wafer.
[0013] The feature extraction module is used to perform multi-scale feature extraction on the first image to obtain the multi-scale features of the first image.
[0014] The encoding module is used to encode the first image and the second image respectively to obtain a first encoded feature corresponding to the first image and a second encoded feature corresponding to the second image.
[0015] The information fusion module is used to fuse the first coding feature and the second coding feature to obtain the third coding feature.
[0016] The generation module is used to generate a feature map of the first image based on the third encoded feature and the multi-scale feature.
[0017] The recognition module is used to identify defects contained in the first image based on the feature map.
[0018] In some implementations, the generation module is configured to: concatenate the third encoded feature and the first encoded feature to obtain a concatenated feature; input the concatenated feature and the multi-scale feature into a decoder, and generate a feature map of the first image through the decoder.
[0019] In some embodiments, the first encoding feature includes a first primitive sequence of the first image, and the second encoding feature includes a second primitive sequence of the second image; the information fusion module is used to perform information fusion on the first primitive sequence and the second primitive sequence to obtain a third primitive sequence, and the third encoding feature includes the third primitive sequence.
[0020] In some implementations, the information fusion module is used to perform cross-attention calculation on the first primitive sequence and the second primitive sequence to obtain the third primitive sequence.
[0021] In some implementations, the information fusion module is used to perform bidirectional cross-attention calculation on the first primitive sequence and the second primitive sequence to obtain a calculation result, the calculation result including a primitive sequence obtained by forward cross-attention calculation and a primitive sequence obtained by reverse cross-attention calculation; and to fuse the primitive sequences obtained by forward cross-attention calculation and the primitive sequences obtained by reverse cross-attention calculation to obtain the third primitive sequence.
[0022] Fourthly, this disclosure provides a model training apparatus, comprising:
[0023] The acquisition module is used to acquire a first image of the wafer to be inspected, a second image of the defect-free wafer, and defect annotation information corresponding to the first image.
[0024] An input module is used to input the first image and the second image into a preset model, and execute the method described in any of the embodiments of the first aspect above through the preset model to obtain the defects contained in the first image.
[0025] The determination module is used to determine the loss value of the preset model based on the defects in the first image and the defect annotation information of the first image.
[0026] The adjustment module is used to adjust the parameters of the preset model based on the loss value until the preset model converges to obtain the target model.
[0027] In some embodiments, the model training apparatus may further include a processing module for performing at least one of the following operations: changing the brightness, contrast, or noise perturbation of the first image and the second image respectively; performing translation processing on the grain image contained in the first image; and fusing non-uniform illumination information in the first image.
[0028] Fifthly, this disclosure also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method of any embodiment of the first aspect or the method of any embodiment of the second aspect.
[0029] In a sixth aspect, this disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of any embodiment of the first aspect or the method of any embodiment of the second aspect.
[0030] In a seventh aspect, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the method of any embodiment of the first aspect or the method of any embodiment of the second aspect.
[0031] The defect detection method, model training method, apparatus, equipment, and medium disclosed herein acquire a first image of a wafer to be inspected and a second image of a defect-free wafer. The first and second images are encoded to obtain a first encoded feature corresponding to the first image and a second encoded feature corresponding to the second image. Information is fused between the first and second encoded features to obtain a third encoded feature, thus realizing feature interaction between the first and second images. The third encoded feature obtained through feature interaction integrates information from both the first and second images, which is beneficial for accurately capturing the differences between them. Furthermore, by performing multi-scale feature extraction on the first image, a feature map of the first image is generated based on the multi-scale features and the third encoded feature. Defect recognition is then performed based on this feature map, which helps improve the sensitivity of defect recognition to local textures and defect boundaries, thereby improving the accuracy of defect recognition. Attached Figure Description
[0032] To more clearly illustrate the technical solutions in the embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the embodiments or related technologies of this disclosure will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0033] Figure 1 This is a flowchart of a defect detection method provided in an embodiment of this disclosure;
[0034] Figure 2 This is a flowchart of an information fusion method provided in an embodiment of this disclosure;
[0035] Figure 3 This is a flowchart of a method for generating a feature map according to an embodiment of this disclosure;
[0036] Figure 4 This is a schematic diagram of a model structure provided in an embodiment of this disclosure;
[0037] Figure 5 This is a flowchart of a model training method provided in an embodiment of this disclosure;
[0038] Figure 6 This is a schematic diagram of the structure of a defect detection device provided in an embodiment of this disclosure;
[0039] Figure 7 This is a schematic diagram of the structure of a model training device provided in an embodiment of this disclosure;
[0040] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0041] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this disclosure.
[0042] It should be noted that the terms "first," "second," etc., used in this disclosure may be used to describe various elements, but these elements are not limited by these terms. These terms are used only to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this disclosure, are intended to cover non-exclusive inclusion. The term "a plurality of" used in this disclosure refers to two or more. The term "and / or" used in this disclosure refers to one of the embodiments, or any combination of multiple embodiments.
[0043] Figure 1This is a flowchart of a defect detection method provided in an embodiment of this disclosure. The method can be executed by a computer device, as exemplarily understood. This computer device can be understood as any device with computing and image processing capabilities, such as a computer, server, distributed computing node, etc., but is not limited to the devices listed herein. Figure 1 As shown, in some embodiments, the defect detection method provided by this disclosure may include the following steps.
[0044] In step S101, a first image of the wafer to be inspected and a second image of the defect-free wafer are acquired.
[0045] In some implementations, the first and second images referred to in the embodiments of this disclosure can be obtained by capturing images with a camera, such as a camera, webcam, or SEM (Scanning Electron Microscope), but are not limited to the devices listed herein.
[0046] The first image may include images of multiple dies / chips. The second image includes a reference image corresponding to each die / chip in the first image.
[0047] In some implementations, after acquiring the first and second images, a preprocessing step may be included. This could include denoising and image alignment, but is not limited to the preprocessing operations listed herein. The denoising methods for the first and second images include, but are not limited to, at least one of the following: Gaussian filtering, mean filtering, median filtering, non-local mean filtering, etc. The image alignment methods for the first and second images include, but are not limited to, at least one of the following: image alignment methods based on ORB (OrientedFAST and Rotated BRIEF, an open-source real-time feature detection and descriptor algorithm), image alignment methods based on keypoint matching, image alignment methods based on phase matching, and dense alignment methods based on optical flow.
[0048] In step S103, multi-scale feature extraction is performed on the first image to obtain the multi-scale features of the first image.
[0049] In some exemplary implementations, a pre-defined CNN (Convolutional Neural Network) can be used to extract multi-scale features of the first image. For example, the pre-defined CNN may include 1x1, 3x3, and 5x5 convolutional kernels. Features of the first image at three scales (1x1, 3x3, and 5x5) are extracted using these 1x1, 3x3, and 5x5 convolutional kernels. The pre-defined CNN network may be, for example, MobileNetV3, but is not limited to MobileNetV3. This is merely an example of multi-scale feature extraction and not a definitive limitation.
[0050] In step S105, the first image and the second image are encoded respectively to obtain the first encoded feature corresponding to the first image and the second encoded feature corresponding to the second image.
[0051] In some examples, the first encoding feature referred to in the embodiments of this disclosure may be specifically a first primitive sequence of a first image (or may also be referred to as a first token sequence), and the second encoding feature may be specifically a second primitive sequence of a second image (or may also be referred to as a second token sequence). Alternatively, in some exemplary statements, it may be stated that the first encoding feature includes the first primitive sequence, and the second encoding feature includes the second primitive sequence.
[0052] For example, in some cases, the first and second images can be divided into n (n is an integer greater than 1) equal-sized image blocks. Each image block is mapped to a D-dimensional embedding feature space through linear projection, resulting in a token sequence for the first image. (i.e., the first primitive sequence) and the token sequence of the second image. (and the second primitive sequence). The first and second primitive sequences include the location code corresponding to the Token. The location code corresponding to the Token indicates the position of the image patch corresponding to that Token within the image.
[0053] In other embodiments, the first coding feature can also be understood as a vector representation of the first image obtained by encoding the entire first image. The second coding feature can also be understood as a vector representation of the second image obtained by encoding the entire second image.
[0054] In step S107, the first coding feature and the second coding feature are fused to obtain the third coding feature.
[0055] In some implementations, information fusion of the first and second coding features can be understood as concatenating the first and second coding features to obtain the third coding feature.
[0056] In other embodiments, information fusion of the first and second coding features can also be understood as obtaining a third coding feature by performing cross-attention calculation on the first and second coding features. For example, when the first coding feature includes a first primitive sequence and the second coding feature includes a second primitive sequence, performing cross-attention calculation on the first and second coding features to obtain the third coding feature can be understood as performing cross-attention calculation on the first and second primitive sequences to obtain the third primitive sequence.
[0057] for example, Figure 2 This is a flowchart of an information fusion method provided in an embodiment of this disclosure. Figure 2 As shown, in some examples, the third primitive sequence can be calculated using the method in steps S201-S203.
[0058] In step S201, bidirectional cross-attention calculation is performed on the first primitive sequence and the second primitive sequence to obtain the calculation result, which includes the primitive sequence obtained by forward cross-attention calculation and the primitive sequence obtained by reverse cross-attention calculation.
[0059] In step S203, the primitive sequences obtained from forward cross-attention calculation and reverse cross-attention calculation are fused to obtain a third primitive sequence.
[0060] For example, suppose the first primitive sequence corresponding to the first image is The second primitive sequence corresponding to the second image is It can and After flattening, there are L layers of cross-attention interaction layers (L is an integer greater than or equal to 1). Each layer has two branches, such as a positive interaction branch that perceives the second image through the first image, and a negative interaction branch that perceives the first image through the second image. In the L layers of cross-attention interaction layers, the output of each positive interaction branch serves as the input to the next positive interaction branch, and the output of each negative interaction branch serves as the input to the next negative interaction branch.
[0061] The reverse interaction branch uses features from the second image to find differences in the first image, enabling the forward interaction branch to better focus on these differences and preventing overfitting. In the i-th (i=0,…,L-1)th cross-attention interaction layer's forward interaction branch, the primitive sequence of the first image input to the previous layer's forward interaction branch is... The primitive sequence of the second image input from the reverse interaction branch of the previous layer is . The query matrix obtained after linear mapping is as follows: , The Key matrix is obtained through linear mapping. and Value Matrix .right , and Calculate the context vector of cross attention :
[0062] (1)
[0063] Then, residual connection and layer normalization operations are performed to obtain the primitive sequence. :
[0064] (2)
[0065] at last By performing a second residual connection through a feedforward network (FFN), the primitive sequence input to the forward interactive branch of the next layer is obtained. :
[0066] (3)
[0067] The handling of reverse interaction branches is similar to that of forward interaction branches. The Query matrix is obtained through linear mapping. . The Key matrix is obtained through linear mapping. and Value Matrix .right , and Calculate the context components of the reverse cross-attention Then, through residual connections, layer normalization, FFN, and layer normalization in sequence, the output of the reverse interaction branch is obtained. After L layers of iteration, the output of the positive interaction branch in the last attention interaction layer is... The feature representation has already been integrated after undergoing multi-layer bidirectional cross-attention. Of course, the above examples are merely illustrative of the cross-attention computation involved in this disclosure, and not the only limitation. In fact, in some examples, unidirectional cross-attention computation can also be used to fuse the information of the first primitive sequence and the second primitive sequence.
[0068] In step S109, a feature map of the first image is generated based on the third coding feature and the multi-scale features of the first image.
[0069] Example, Figure 3 This is a flowchart of a method for generating a feature map according to an embodiment of this disclosure. Figure 3 As shown, in some examples, the feature map of the first image can be generated based on the method of steps S301-S303.
[0070] In step S301, the third coding feature and the first coding feature are concatenated to obtain the concatenated feature.
[0071] For example, the third coding feature and the first coding feature can be concatenated along the channel dimension to obtain the concatenated feature.
[0072] In step S303, the spliced features and the multi-scale features of the first image are input into the decoder, and the feature map of the first image is generated by the decoder.
[0073] In some examples, the stitched features and multi-scale features of the first image can be directly input into the decoder to generate a feature map of the first image based on the decoder.
[0074] In other examples, the stitched features can be input into a preset M-layer Transformer encoder (M being an integer greater than or equal to 1). The M-layer Transformer encoder outputs the encoded features corresponding to the stitched features. In this case, the encoded features of the stitched features and the multi-scale features of the first image can be input into the decoder, which generates a feature map of the first image based on the encoded features of the stitched features and the multi-scale features of the first image.
[0075] certainly Figure 3 This is merely an exemplary method for generating feature maps, not the only one. In fact, in other implementations, the third encoded feature and the multi-scale features of the first image can be directly input into a preset decoder, which then generates a feature map of the first image based on the third encoded feature and the multi-scale features of the first image.
[0076] In step S111, defects contained in the first image are identified based on the feature map of the first image.
[0077] For example, the feature map of the first image can be input into a preset prediction head, which can then identify the defects contained in the first image.
[0078] according to Figure 1In this embodiment of the present disclosure, a first image of a wafer to be inspected and a second image of a defect-free wafer are acquired. The first and second images are then encoded to obtain a first encoded feature corresponding to the first image and a second encoded feature corresponding to the second image. Information fusion is performed on the first and second encoded features to obtain a third encoded feature. This achieves feature interaction between the first and second images. The third encoded feature obtained through feature interaction fuses information from the first and second images, which is beneficial for accurately capturing the differences between the first and second images. By performing multi-scale feature extraction on the first image, a feature map of the first image is generated based on the multi-scale features and the third encoded feature. Defect recognition is performed based on the feature map of the first image, which is beneficial for improving the sensitivity of defect recognition to local textures and defect boundaries, and improving the accuracy of defect recognition.
[0079] Figure 4 This is a schematic diagram of a model structure provided in an embodiment of this disclosure. For example... Figure 4 As shown, the model may include a cross-image interaction module 410, a Transformer encoder 420, a MobileNetV3 network 430, a decoder 440, and a prediction head 450. The cross-image interaction module 410 performs cross-attention calculation on a first image of the wafer to be inspected and a second image of a defect-free wafer, outputting a primitive sequence that fuses information from both images. The Transformer encoder 420 encodes the primitive sequence output by the cross-image interaction module 410. The MobileNetV3 network 430 extracts multi-scale features from the first image. The decoder 440 generates a feature map of the first image based on the outputs of the MobileNetV3 network 430 and the Transformer encoder 420. The prediction head 450 generates a defect distribution matrix of the first image based on the feature map, the defect distribution matrix including defect location information and the corresponding confidence / probability.
[0080] Based on this, some embodiments of this disclosure also provide a model training method, and through this method, a model training method is provided. Figure 4 The model shown is trained. The specific method includes:
[0081] S1. Collect wafers that can provide defect data, obtain an image of each wafer (i.e., the first image) and its corresponding image of a defect-free wafer (i.e., the second image), and construct an original image pair database consisting of the first image and the second image.
[0082] S2. Iterate through each pair of images in the database and apply a Gaussian filter with a kernel of 3 to smooth all images, reducing image noise. Then, perform feature point matching based on ORB to find the spatial transformation relationship between the first and second images, achieving image alignment.
[0083] S3. Traverse the original images for each pair of images in the database, and calculate the first image. Second image Difference diagram And use a preset threshold T to analyze the difference map Binarize the pixels on the map to obtain a binary map of the defect distribution. In this context, a pixel value of 255 indicates a defect, and a pixel value of 0 indicates no defect.
[0084] (4)
[0085] (5)
[0086] S4. Use the boundary tracing algorithm to analyze the binary map of defect distribution. Perform contour fitting to obtain a set of several contour points. Each set of contour points encloses a local region representing the location information of a single potential defect in the first image. For each first image, all sets of contour points are recorded, representing all its potential defect information.
[0087] S5. Convert the potential defect information of the first image obtained in S4 into the labeling format of the image annotation tool LabelMe, and import it into LabelMe along with the first image for manual review. The reviewers will examine each image to determine if the defect information is accurate. If it is not a defect, it will be deleted from the potential defect information; if the defect location is inaccurate, the defect location will be adjusted, and finally, the defect annotation information used for the first image will be obtained.
[0088] S6. The defect annotation information after manual review is converted into a defect distribution matrix G in a specified format. In matrix G, the location of a defect is set to 1, and the location of no defect is set to 0. Specifically, a matrix of all zeros with the same size as the first image is created. Each pixel in the first image is iterated over; if the pixel's location is inside a potential defect in the annotation, the element at that location in the matrix is set to 1; otherwise, it is set to 0. The first image, the second image, and the defect distribution moments of the first image are combined to form the training data.
[0089] S7. Training phase, the first image Second image Divide into sizes of The image patches are mapped to a D-dimensional embedding feature space through a learnable linear projection, resulting in a token sequence for the first image. Token sequence of the second image Furthermore, a learnable location code is added to each token to provide spatial location information.
[0090] S8. Extract the token sequence from S7. and Flattened input into the cross-image interaction module Each layer employs a cross-attention interaction layer, with two branches: a forward interaction branch that uses the first image to perceive the second image, and a backward interaction branch that uses the second image to perceive the first image. The backward interaction branch uses features from the second image to find differences in the first image, indirectly constraining the forward interaction branch to better focus on the differences between the first and second images, thus preventing overfitting. In the... layer In the forward interaction branch of the cross-attention interaction layer, the primitive sequence of the first image input from the forward interaction branch of the previous layer is: The primitive sequence of the second image input from the reverse interaction branch of the previous layer is: .enter The Query matrix is obtained through linear mapping. , The Key matrix is obtained through linear mapping. and Value Matrix .right , and Calculate the context vector of cross attention Subsequently, residual connections and layer normalization are performed to obtain the primitive sequence. .
[0091] at last The residuals are then reconnected through a feedforward network (FFN) to obtain the forward input for the next layer. .
[0092] The reverse interaction branch works similarly to the forward interaction branch. The Query matrix is obtained through linear mapping. , The Key matrix is obtained through linear mapping. and Value Matrix .right , and Calculate the context components of the reverse cross-attention Then, through residual connections, layer normalization, FFN, and layer normalization in sequence, the output of the reverse interaction branch is obtained. After L layers of iteration, the output of the positive interaction branch in the last attention interaction layer is... The feature representation, after undergoing multi-layer bidirectional cross-attention, has been integrated. During training, to address interference factors such as illumination variations, non-uniform illumination, and alignment errors that may occur in the wafer image, random data augmentation is used to enable the cross-image interaction module 410 to obtain highly robust fused features. Specific data augmentation options include: applying independent brightness, contrast, and noise perturbations to the first and second images respectively to simulate illumination variations that may occur in the actual wafer image; applying a small random translation to the first image to simulate alignment errors that may occur in the actual wafer image; generating an illumination gain map with a quadratic or Gaussian radial decay distribution, and randomly multiplying the pixel values of the first and second images point-by-point with the gain map to simulate the wafer image under the influence of non-uniform illumination.
[0093] S9. The token sequence output by the cross-image interaction module 410 and the token sequence of the first image are concatenated along the channel dimension to obtain... .Will The input is an M-layer Transformer encoder 420, each layer containing a multi-head self-attention network and a feedforward network. The output of the M-layer encoder yields the encoded features. This is used for subsequent defect detection and decoding.
[0094] S10. Encoding features Perform a 1x1 convolutional layer, adjust the channels, and rearrange them according to the original spatial arrangement to form two-dimensional encoded features. Additionally, a MobileNetV3 network 430 was used to extract features from the first image, obtaining multi-scale features containing different semantic levels. Two-dimensional encoding features The multi-scale features extracted from the MobileNetV3 network 430 are input into the decoder 440 for progressive upsampling decoding. In each stage of upsampling decoding, the two-dimensional encoded features are first processed... Bilinear upsampling is performed to match the spatial dimensions of the features at the corresponding level. Then, the upsampled features are concatenated with the features at the corresponding scale along the channel dimension. A 3x3 convolution and a non-linear activation function are sequentially applied to the concatenated features to obtain the fused features. After progressively upsampling and convolutional processing of the fused features, the final output restores a multi-channel two-dimensional feature map with the same resolution as the first image. .Will Input prediction head 450, which consists of a 1x1 convolutional layer and a sigmoid activation function, is used to process multi-channel two-dimensional feature maps. Mapped to a single-channel defect distribution map During the training phase, the defect distribution map... The true defect distribution matrix of the first image Calculate the binary cross-entropy loss and backpropagate it to calculate the gradient and update the network parameters until the loss function converges, completing model training. During the inference phase, input the first and second images to the trained model. The model outputs a defect distribution matrix. Elements with values greater than 0.5 in the matrix indicate the presence of a defect at the corresponding location in the first image, while elements with values less than 0.5 indicate the absence of a defect at the corresponding location in the first image.
[0095] (6)
[0096] in, The coordinates on the first image are element at position The probability of defects.
[0097] It should be noted that the above-described model training method is merely an illustrative example of the model training method provided in this disclosure, and not the only limitation. In fact, in other embodiments, the model training method provided in this disclosure can be exemplarily described as follows: Figure 5 The process shown in steps S501-S505. Alternatively, in some exemplary descriptions, the model training method provided in this embodiment can be generally described as acquiring a first image of the wafer to be inspected, a second image of the defect-free wafer, and defect annotation information of the first image; inputting the first image and the second image into a preset model, and performing the above-described steps through the preset model. Figures 1-3 The method described in any embodiment of the present invention obtains defects contained in a first image; determines a loss value for a preset model based on the defects in the first image and the defect annotation information of the first image; and adjusts the parameters of the preset model based on the loss value until the preset model converges to obtain a target model. Before inputting the first image and the second image into the preset model, the model training method may further include at least one of the following operations: changing the brightness, contrast, or noise perturbation of the first image and the second image respectively; performing translation processing on the grain image contained in the first image; and fusing non-uniform illumination information in the first image.
[0098] This embodiment simplifies the tedious manual pixel-by-pixel defect labeling process by using a semi-automatic process of "automatic initial screening using differential method + manual re-screening", which greatly reduces the manpower and time costs of obtaining large-scale, high-quality defect labeling data.
[0099] The cross-image interaction module in this embodiment uses bidirectional cross-attention to interact with the first and second images. Through iterative training and optimization of the interaction features, the features can more accurately capture the differences between the first and second images. The targeted data augmentation strategy in the cross-image interaction module also significantly enhances its generalization ability and stability against common interferences in the wafer image domain.
[0100] The hybrid architecture of CNN and Transformer is adopted. The Transformer architecture in the cross-image interaction module and encoder can ensure accurate modeling of the difference relationship between the first image and the second image. The CNN multi-scale feature extraction used in the decoder ensures the sensitivity of defect recognition to local texture and defect boundaries, thereby improving the accuracy and robustness of defect detection.
[0101] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this disclosure.
[0102] Based on the same inventive concept, this disclosure also provides a defect detection apparatus for implementing the defect detection method described above, and a model training apparatus for implementing the model training method described above. The solutions provided by these apparatuses are similar to the solutions described in the above methods; therefore, specific limitations in one or more apparatus embodiments provided below can be found in the limitations of the corresponding methods described above, and will not be repeated here.
[0103] Figure 6 This is a schematic diagram of the structure of a defect detection device provided in an embodiment of this disclosure. Figure 6 As shown, in some exemplary embodiments, the defect detection device 600 provided in this disclosure includes:
[0104] The acquisition module 601 is used to acquire a first image of the wafer to be inspected and a second image of the defect-free wafer.
[0105] The feature extraction module 602 is used to perform multi-scale feature extraction on the first image to obtain the multi-scale features of the first image.
[0106] The encoding module 603 is used to encode the first image and the second image respectively to obtain a first encoded feature corresponding to the first image and a second encoded feature corresponding to the second image.
[0107] The information fusion module 604 is used to fuse the first coding feature and the second coding feature to obtain the third coding feature.
[0108] The generation module 605 is used to generate a feature map of the first image based on the third coding feature and the multi-scale feature.
[0109] The recognition module 606 is used to identify defects contained in the first image based on the feature map.
[0110] In some implementations, the generation module 605 is configured to: concatenate the third encoded feature and the first encoded feature to obtain a concatenated feature; input the concatenated feature and the multi-scale feature into a decoder, and generate a feature map of the first image through the decoder.
[0111] In some embodiments, the first encoding feature includes a first primitive sequence of the first image, and the second encoding feature includes a second primitive sequence of the second image; the information fusion module 604 is used to perform information fusion on the first primitive sequence and the second primitive sequence to obtain a third primitive sequence, wherein the third encoding feature includes the third primitive sequence.
[0112] In some implementations, the information fusion module 604 is used to perform cross-attention calculation on the first primitive sequence and the second primitive sequence to obtain the third primitive sequence.
[0113] In some implementations, the information fusion module 604 is used to perform bidirectional cross-attention calculation on the first primitive sequence and the second primitive sequence to obtain a calculation result, the calculation result including a primitive sequence obtained by forward cross-attention calculation and a primitive sequence obtained by reverse cross-attention calculation; and to fuse the primitive sequences obtained by forward cross-attention calculation and reverse cross-attention calculation to obtain the third primitive sequence.
[0114] Figure 7 This is a schematic diagram of the structure of a model training device provided in an embodiment of this disclosure. For example... Figure 7 As shown, in some examples, the model training apparatus 700 provided in this disclosure embodiment may include:
[0115] The acquisition module 701 is used to acquire a first image of the wafer to be inspected, a second image of the defect-free wafer, and defect annotation information corresponding to the first image.
[0116] The input module 702 is used to input the first image and the second image into a preset model, and execute the method described in any of the embodiments of the first aspect above through the preset model to obtain the defects contained in the first image.
[0117] The determination module 703 is used to determine the loss value of the preset model based on the defects in the first image and the defect annotation information of the first image.
[0118] The adjustment module 704 is used to adjust the parameters of the preset model based on the loss value until the preset model converges to obtain the target model.
[0119] In some embodiments, the model training apparatus 700 may further include a processing module for performing at least one of the following operations: changing the brightness, contrast, or noise perturbation of the first image and the second image respectively; performing translation processing on the grain image contained in the first image; and fusing non-uniform illumination information in the first image.
[0120] The aforementioned defect detection device and model training device, as well as each module, can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.
[0121] In one exemplary embodiment, a computer device is provided, which may be a computer or a server, and its internal structure diagram may be as follows: Figure 8As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data for the first and second images. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a defect detection method or a model training method.
[0122] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the computer device to which the present disclosure is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0123] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the defect detection method or model training method in any of the above method embodiments.
[0124] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the defect detection method or model training method in any of the above method embodiments.
[0125] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the defect detection method or model training method in any of the above method embodiments.
[0126] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, databases, or other media used in the embodiments provided in this disclosure can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this disclosure may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this disclosure may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.
[0127] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this disclosure.
[0128] The embodiments described above are merely illustrative of several implementations of this disclosure, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent disclosure. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this disclosure, and these all fall within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the appended claims.
Claims
1. A defect detection method, characterized in that, The method includes: Acquire a first image of the wafer to be inspected, and a second image of the defect-free wafer; Multi-scale feature extraction is performed on the first image to obtain the multi-scale features of the first image; The first image and the second image are encoded respectively to obtain a first encoded feature corresponding to the first image and a second encoded feature corresponding to the second image; The first and second coding features are fused to obtain the third coding feature; Based on the third coding feature and the multi-scale feature, a feature map of the first image is generated; Defects contained in the first image are identified based on the feature map.
2. The method according to claim 1, characterized in that, The step of generating a feature map of the first image based on the third encoded feature and the multi-scale feature includes: The third coding feature and the first coding feature are concatenated to obtain the concatenated feature; The stitched features and the multi-scale features are input into the decoder, and the feature map of the first image is generated by the decoder.
3. The method according to claim 1 or 2, characterized in that, The first coding feature includes a first primitive sequence of the first image, and the second coding feature includes a second primitive sequence of the second image; The process of fusing the first and second coding features to obtain the third coding feature includes: Information fusion is performed on the first primitive sequence and the second primitive sequence to obtain a third primitive sequence, and the third coding feature includes the third primitive sequence.
4. The method according to claim 3, characterized in that, The step of fusing information from the first primitive sequence and the second primitive sequence to obtain the third primitive sequence includes: Cross-attention calculation is performed on the first primitive sequence and the second primitive sequence to obtain the third primitive sequence.
5. The method according to claim 4, characterized in that, The step of performing cross-attention calculation on the first primitive sequence and the second primitive sequence to obtain the third primitive sequence includes: A bidirectional cross-attention calculation is performed on the first primitive sequence and the second primitive sequence to obtain a calculation result. The calculation result includes a primitive sequence obtained by forward cross-attention calculation and a primitive sequence obtained by reverse cross-attention calculation. The primitive sequences obtained from the forward cross-attention calculation and the reverse cross-attention calculation are fused to obtain the third primitive sequence.
6. A model training method, characterized in that, include: Acquire a first image of the wafer to be inspected, a second image of the defect-free wafer, and defect annotation information of the first image; The first image and the second image are input into a preset model, and the method described in any one of claims 1-5 is executed through the preset model to obtain the defects contained in the first image; Based on the defects in the first image and the defect annotation information of the first image, the loss value of the preset model is determined; Based on the loss value, the parameters of the preset model are adjusted until the preset model converges to obtain the target model.
7. The method according to claim 6, characterized in that, Before inputting the first image and the second image into the preset model, the method further includes at least one of the following operations: The brightness, contrast, or noise perturbation of the first and second images are changed respectively. The grain image contained in the first image is translated. Non-uniform lighting information is fused into the first image.
8. A defect detection device, characterized in that, include: The acquisition module is used to acquire a first image of the wafer to be inspected and a second image of the defect-free wafer; The feature extraction module is used to perform multi-scale feature extraction on the first image to obtain the multi-scale features of the first image; The encoding module is used to encode the first image and the second image respectively to obtain a first encoded feature corresponding to the first image and a second encoded feature corresponding to the second image; An information fusion module is used to fuse the first coding feature and the second coding feature to obtain a third coding feature; The generation module is used to generate a feature map of the first image based on the third encoded feature and the multi-scale feature; The recognition module is used to identify defects contained in the first image based on the feature map.
9. A model training device, characterized in that, include: The acquisition module is used to acquire a first image of the wafer to be inspected, a second image of the defect-free wafer, and defect annotation information corresponding to the first image; An input module is used to input the first image and the second image into a preset model, and execute the method as described in any one of claims 1-5 through the preset model to obtain the defects contained in the first image; The determination module is used to determine the loss value of the preset model based on the defects in the first image and the defect annotation information of the first image; The adjustment module is used to adjust the parameters of the preset model based on the loss value until the preset model converges to obtain the target model.
10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5, or 6 to 7.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5, or 6 to 7.
12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5, or 6 to 7.