Wafer mixed defect detection method and system based on multimodal deep learning

Abstract

The present invention provides a wafer hybrid defect detection method and system based on multimodal deep learning, the method comprising: acquiring a multimodal image of a wafer; wherein the multimodal image comprises a brightfield image, a darkfield image and a differential interference contrast image; normalizing the multimodal image; performing channel stitching on the normalized multimodal image to obtain a multi-channel input image; based on a dual-branch deep learning network, detecting the defect type, defect quantity, defect bounding box coordinates, a first probability distribution of the defect type and a second probability distribution of the defect quantity of the multi-channel input image; performing weighted fusion of the first probability distribution and the second probability distribution to obtain a defect detection result, which can effectively improve the defect detection accuracy and significantly reduce the missed detection rate, while solving the problem of insufficient recognition ability of traditional single-modal detection methods in complex backgrounds.

Description

Technical Field

[0001] The present invention relates to the technical field of wafer defect detection, and in particular to a wafer mixed defect detection method and system based on multimodal deep learning. Background Art

[0002] In the semiconductor manufacturing process, wafer defect detection is a key link in ensuring product quality and production efficiency.

[0003] Traditional wafer defect detection methods mainly rely on single-modality imaging technologies, such as brightfield imaging or darkfield imaging. However, these single-modality detection methods have many limitations. For example, although brightfield imaging can effectively detect defects such as particles and dirt with obvious differences in surface reflectivity, its recognition ability is weak for defects on the back of transparent wafers and mixed defects in some complex backgrounds; darkfield imaging has a good detection effect on defects with prominent edge features such as scratches and chipped edges, but has low detection sensitivity for surface defects such as particles. In addition, the classification accuracy of defects in existing technologies is generally low, especially for rare defects. Due to insufficient sample size, the model generalization ability is poor and the missed detection rate is higher. Summary of the Invention

[0004] The present invention provides a wafer hybrid defect detection method and system based on multimodal deep learning, which are used to solve the technical problem of low wafer defect detection accuracy in the prior art.

[0005] In one aspect, the present invention provides a wafer mixed defect detection method based on multimodal deep learning, comprising:

[0006] Acquire a multimodal image of the wafer; wherein the multimodal image includes a bright field image, a dark field image, and a differential interference contrast image;

[0007] Normalizing the multimodal image;

[0008] Perform channel splicing on the normalized multimodal image to obtain a multi-channel input image;

[0009] Based on a dual-branch deep learning network, detect the defect type, defect quantity, defect bounding box coordinates, a first probability distribution of the defect type, and a second probability distribution of the defect quantity of the multi-channel input image;

[0010] The first probability distribution and the second probability distribution are weightedly fused to obtain a defect detection result.

[0011] According to a wafer hybrid defect detection method based on multimodal deep learning provided by the present invention, the step of acquiring a multimodal image of a wafer includes:

[0012] illuminating the surface of the wafer with direct light to obtain the bright field imaging;

[0013] Oblique light is used to illuminate the surface of the wafer to obtain the dark field image;

[0014] The differential interference contrast image is obtained by illuminating the surface of the wafer with polarized light.

[0015] According to a wafer hybrid defect detection method based on multimodal deep learning provided by the present invention, the dual-branch deep learning network includes a first branch and a second branch. The dual-branch deep learning network detects the defect type, defect quantity, defect bounding box coordinates, the first probability distribution of the defect type, and the second probability distribution of the defect quantity of the multi-channel input image, including:

[0016] Inputting the multi-channel input image into the lightweight ResNet network of the first branch, extracting texture features and geometric features of the defect, and generating a feature vector;

[0017] Performing global average pooling on the feature vector to obtain a dimension-reduced feature vector;

[0018] Input the reduced dimension feature vector into a fully connected layer, and output the defect type and the corresponding first probability distribution through a Sigmoid activation function;

[0019] Inputting the multi-channel input image into the basic convolution of the second branch to extract initial features of the multi-channel input image;

[0020] Based on the initial features, output the number of defects and the corresponding second probability distribution;

[0021] reshape the second probability distribution of the number of defects into a feature map to characterize the spatial distribution of the number of defects;

[0022] The initial multimodal image is input into the second branch, and the spatial features of the multimodal image are extracted through multi-layer convolution operations;

[0023] Splicing the spatial features with the feature map to obtain a feature splicing map;

[0024] The feature mosaic image is detected to obtain the coordinates of the defect boundary box.

[0025] According to a wafer hybrid defect detection method based on multimodal deep learning provided by the present invention, the first probability distribution and the second probability distribution are weightedly fused to obtain a defect detection result, including:

[0026] Dynamically adjust the weights of the first and second branches according to the number of defects;

[0027] fusing the first probability distribution of the defect type and the second probability distribution of the defect quantity according to the weights of the first branch and the second branch to obtain a fusion result;

[0028] The fusion results are optimized using the joint loss function to obtain the defect detection results.

[0029] According to a wafer mixed defect detection method based on multimodal deep learning provided by the present invention, the weights of the first branch and the second branch are dynamically adjusted according to the number of defects, including:

[0030] ;

[0031] in, is the weight of the first branch, n is the number of defects, k is the adjustment factor, and e is the base of the natural logarithm;

[0032] ;

[0033] in, is the weight of the second branch.

[0034] According to a wafer hybrid defect detection method based on multimodal deep learning provided by the present invention, the use of a joint loss function to optimize the fusion result to obtain a defect detection result includes:

[0035] ;

[0036] in, Loss is the joint loss function, is the classification cross entropy loss, is the GIoU positioning loss, is the mean square error loss of the number of defects, for The coefficient of for The coefficient of .

[0037] A wafer mixed defect detection method based on multimodal deep learning provided by the present invention also includes:

[0038] Generate an analysis report based on the pre-established mapping relationship between defect detection results and process parameters; wherein the analysis report includes defect cause analysis, tuning suggestions and expected results.

[0039] According to a wafer hybrid defect detection method based on multimodal deep learning provided by the present invention, before acquiring a multimodal image of the wafer, the method further includes:

[0040] Plasma treatment is used to clean and modify the wafer surface, so that a uniform nano-scale texture structure is formed on the wafer surface.

[0041] A wafer mixed defect detection method based on multimodal deep learning provided by the present invention also includes:

[0042] According to the defect detection results, the defects on the wafer are marked using quantum dot fluorescent marking technology.

[0043] On the other hand, the present invention also provides a wafer hybrid defect detection system based on multimodal deep learning, comprising:

[0044] An image acquisition module, configured to acquire a multimodal image of the wafer; wherein the multimodal image includes a bright field image, a dark field image, and a differential interference contrast image;

[0045] A preprocessing module, configured to perform normalization processing on the multimodal image;

[0046] The channel stitching module is used to stitch the normalized multimodal images into channels to obtain a multi-channel input image;

[0047] a defect analysis module, configured to detect the defect type, the number of defects, the coordinates of the defect bounding box, the first probability distribution of the defect type, and the second probability distribution of the number of defects of the multi-channel input image based on a dual-branch deep learning network;

[0048] The defect determination module is used to perform weighted fusion of the first probability distribution and the second probability distribution to obtain a defect detection result.

[0049] The wafer mixed defect detection method and system based on multimodal deep learning provided by the present invention performs channel stitching on the normalized multimodal image to obtain a multi-channel input image; based on a dual-branch deep learning network, detects the defect type, defect quantity, defect bounding box coordinates, first probability distribution of defect type and second probability distribution of defect quantity of the multi-channel input image; performs weighted fusion of the first probability distribution and the second probability distribution to obtain a defect detection result, which can effectively improve the detection accuracy of defects (such as mixed defects), significantly reduce the missed detection rate, and at the same time solve the problem of insufficient recognition ability of traditional single-modal detection methods in complex backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 11 is a flow chart of a wafer hybrid defect detection method based on multimodal deep learning provided by an embodiment of the present invention;

[0052] Figure 2 1 is a schematic structural diagram of a wafer hybrid defect detection system based on multimodal deep learning provided by an embodiment of the present invention;

[0053] Figure 3 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0055] Figure 1 This is a flow chart of a hybrid wafer defect detection method based on multimodal deep learning, as provided by an embodiment of the present invention. This method is applicable to front- and back-side defect detection on wafers made of materials such as silicon and glass (e.g., 4-12 inches).

[0056] See also Figure 1 , the wafer mixed defect detection method based on multimodal deep learning may include the following steps 101 to 105.

[0057] Step 101: Acquire a multimodal image of a wafer; wherein the multimodal image includes a bright field image, a dark field image, and a differential interference contrast image.

[0058] Step 101 may specifically include:

[0059] Bright Field (BF) imaging is obtained by illuminating the surface of the wafer with direct light.

[0060] Oblique light is used to illuminate the surface of the wafer to obtain a dark field (DF) image.

[0061] Polarized light is used to illuminate the surface of the wafer to obtain a differential interference contrast (DIC) image.

[0062] In this step, for bright field imaging: the light emitted by the light source directly illuminates the wafer surface, and the image is formed by the brightness difference between the defect and the background. This can highlight defects with obvious differences in surface reflectivity, such as particles and dirt.

[0063] For dark field images: the light emitted by the light source is obliquely incident on the wafer surface at a certain angle, mainly capturing the scattered light at the edge of the defect. This imaging method can enhance the contour features of the defect and is suitable for detecting defects with prominent edge features such as scratches and chipped edges.

[0064] For differential interference contrast images: a three-dimensional relief image is generated through polarized light interference technology. After the polarized light emitted by the light source passes through the wafer surface, the phase of the light will change due to the presence of defects. The interference technology can generate an image that highlights the difference in defect height and texture.

[0065] By acquiring brightfield, darkfield, and differential interference contrast (DIC) images, this method can comprehensively capture wafer defect characteristics from multiple angles. Brightfield images provide information on surface reflectivity differences, darkfield images highlight defect contours, and DIC images reveal the three-dimensional structure of subsurface defects. This multimodal data fusion approach can significantly improve the detection accuracy of complex defects (such as mixed defects), overcoming the limitations of traditional single-modal inspection methods.

[0066] The specific implementation can be as follows:

[0067] Optical imaging system: Equipped with 1X, 2X, 5X, 10X, 20X, and 50X adjustable magnification lenses, the electric zoom mechanism enables 0.5μm-10μm resolution switching to meet the detection needs of macro defects (such as scratches) and micro defects (such as nano-scale particles);

[0068] Mechanical and control unit: A vacuum adsorption fixture supports contactless clamping of the wafer backside, preventing contamination of the active backside area and ensuring stable positioning of warped wafers (warpage ≤ 50μm). Furthermore, a laser ranging sensor and servo motor enable real-time autofocus with a focus time of ≤ 20ms, ensuring clear imaging of wafers of varying thicknesses (0.1mm-1mm).

[0069] Time Delay Integration (TDI) Camera: Combined with a high-power LED linear array light source, it achieves an 8-inch wafer inspection efficiency of ≥12 wafers / hour (resolution 1μm), reduces the impact of motion blur, and improves image acquisition speed.

[0070] Step 102: normalize the multimodal image.

[0071] In this step, multimodal images are normalized to bring the image data from different modalities to the same scale, eliminating differences between the data and improving model training and detection accuracy. Brightfield, darkfield, and differential interference contrast images can be normalized to [-1, 1].

[0072] In addition to normalizing multimodal images, noise suppression can also be performed on multimodal images, for example, by performing median filtering and wavelet denoising. Median filtering is a nonlinear filtering technique that reduces noise by replacing each pixel value with the median value within its neighborhood. This method is effective for removing random noise such as salt and pepper noise, while also preserving the image's edge information. Wavelet denoising is a noise suppression method based on wavelet transforms. It removes noise by decomposing the image into subbands of different frequencies and then performing thresholding on the high-frequency subbands. Wavelet denoising can effectively preserve image details and structural information while removing noise, making it suitable for processing complex image noise.

[0073] To address the class imbalance problem, a generative adversarial network (GAN) is used to generate rare defect samples (such as edge collapse and nearly full defects). Manifold regularization is used to keep the generated samples consistent with the manifold structure of the real data to avoid pattern distortion. Image data can also be enhanced. For example, conventional enhancement operations may include: random rotation (±15°), Gaussian noise addition ( ), local occlusion (simulating wafer edge occlusion scenarios), etc.

[0074] Step 103: perform channel stitching on the normalized multimodal image to obtain a multi-channel input image.

[0075] In this step, if the image of each modality is 3-channel (color image), they can be stitched together into a 9-channel input image. For example, 3 channels of brightfield image, 3 channels of darkfield image, and 3 channels of differential interference contrast image together constitute a 9-channel input image. Through channel stitching, the multi-channel input image can simultaneously contain the feature information of brightfield, darkfield, and differential interference contrast images. This multimodal data fusion method can significantly improve the feature expression ability of complex defects (such as mixed defects), enabling subsequent deep learning networks to more accurately detect and classify defects. Images of different modalities provide defect features captured from different angles. By fusing these features together, the shortcomings of single-modality images in certain defect detection can be compensated, thereby improving the overall accuracy of defect detection.

[0076] Step 104: Based on the dual-branch deep learning network, detect the defect type, defect quantity, defect bounding box coordinates, first probability distribution of defect type, and second probability distribution of defect quantity of the multi-channel input image.

[0077] Specifically, the dual-branch deep learning network may include a first branch and a second branch. Step 104 may include:

[0078] Input the multi-channel input image into the lightweight ResNet network of the first branch to extract the texture features and geometric features of the defect and generate a feature vector;

[0079] Perform global average pooling on the feature vector to obtain a reduced-dimensional feature vector;

[0080] The reduced dimension feature vector is input into the fully connected layer, and the defect type and the corresponding first probability distribution are output through the Sigmoid activation function;

[0081] Input the multi-channel input image into the basic convolution of the second branch to extract the initial features of the multi-channel input image;

[0082] Based on the initial features, output the number of defects and the corresponding second probability distribution;

[0083] reshape the second probability distribution of the number of defects into a feature map to characterize the spatial distribution of the number of defects;

[0084] The initial multimodal image is input into the second branch, and the spatial features of the multimodal image are extracted through multi-layer convolution operations;

[0085] Splice the spatial features with the feature map to obtain a feature splicing map;

[0086] The feature mosaic image is detected to obtain the coordinates of the defect bounding box.

[0087] The following describes the first and second branches in detail. Generally, the first branch uses a lightweight ResNet network (such as ResNet-34) as the backbone network. This network extracts texture and geometric features of the image through multiple residual blocks (e.g., four). Each residual block contains two 3×3 convolutional layers. This effectively extracts texture (e.g., grain roughness) and geometric (e.g., scratch length and angle) features of the defect, generating a high-dimensional feature vector (e.g., 512-dimensional). The reduced feature vector can be 256-dimensional. The reduced feature vector is input to a fully connected layer, which uses a Sigmoid activation function to output the defect type and its corresponding first probability distribution. The Sigmoid function maps the output value to the (0, 1) interval, representing the probability of each defect type. For example, an output value of 0.9 indicates a 90% probability of the presence of that defect type. Defect types can include center defects, circular ring defects, edge ring defects, and so on. In addition, the defect type can be refined by concatenating the preliminary judgment results with the shallow features (16×16×256) of ResNet-34. Through three fully connected layers (256 nodes per layer) and the Softmax function, the probability distribution of 50 defect types (including single defects and mixed defects) can be output.

[0088] The second branch extracts initial features through basic convolution (Basic Block2, which consists of two layers of 3×3 convolution, batch normalization, and ReLU). It then outputs a probability distribution of the number of defects using Softmax. Feature concatenation and an improved YOLOv4 detection head are then used to output the bounding box coordinates of the defects. Basic convolutional layers typically consist of multiple layers of convolution operations. Each convolution layer uses a set of convolution filters to slide over the input image to extract local features. For example, a basic convolutional layer might contain two layers of 3×3 convolution operations. Initial features extracted by the basic convolutional layers contain basic feature information of the input image, such as edges, texture, and shape.

[0089] The second probability distribution of the number of defects is reshaped into a feature map. This feature map represents the spatial distribution of the number of defects and can intuitively display the location and number of defects in the image. For example, the defect count result is embedded and reshaped into a 4×4×16 feature map, where 4×4 represents the feature map size and 16 represents the number of channels. The spatial features (extracted from the image using Basic Block3 (containing 5 convolution layers and stride 2 downsampling)) are concatenated with the feature map to produce a concatenated feature map. This concatenation operation combines the spatial distribution of the number of defects with the spatial features, providing richer information for defect localization. The coordinates of the defect bounding box may include the center position, width, and height of the defect. This process usually uses an improved version of the YOLOv4 detection head (the detection head contains three detection scales (13×13, 26×26, and 52×52). Each scale predicts the defect bounding box and category confidence to solve the problem of misjudgment of large-sized pollutants, such as avoiding identifying a single large particle as multiple small particles). This detection head improves the speed and accuracy of defect positioning through multi-scale feature fusion and anchor frame optimization.

[0090] Step 105: Perform weighted fusion on the first probability distribution and the second probability distribution to obtain a defect detection result.

[0091] In this embodiment, the normalized multimodal image is channel-stitched to obtain a multi-channel input image; based on a dual-branch deep learning network, the defect type, defect quantity, defect bounding box coordinates, first probability distribution of defect type and second probability distribution of defect quantity of the multi-channel input image are detected; the first probability distribution and the second probability distribution are weightedly fused to obtain a defect detection result, which can effectively improve the detection accuracy of defects (such as mixed defects), significantly reduce the missed detection rate, and solve the problem of insufficient recognition ability of traditional single-modal detection methods in complex backgrounds.

[0092] In one embodiment of this specification, performing weighted fusion on the first probability distribution and the second probability distribution to obtain a defect detection result includes:

[0093] Dynamically adjust the weights of the first and second branches according to the number of defects;

[0094] fusing the first probability distribution of the defect type and the second probability distribution of the defect quantity according to the weights of the first branch and the second branch to obtain a fusion result;

[0095] The fusion results are optimized using the joint loss function to obtain the defect detection results.

[0096] In this embodiment, the fusion result is a comprehensive probability distribution, representing the model's confidence in different combinations of defect type and quantity. Based on this fusion result, a final defect detection decision can be made. For example, the defect type and quantity combination with the highest confidence score is selected as the final detection result. This process considers both defect type and quantity, and by assigning appropriate weights, the final result more comprehensively reflects the defect situation on the wafer.

[0097] It is understandable that the dual-branch deep learning network is pre-trained and can be trained, verified, and tested using a large number of data samples and defect samples.

[0098] In one embodiment of the present specification, the weights of the first branch and the second branch are dynamically adjusted according to the number of defects, including the following formulas (1) and (2):

[0099] (1);

[0100] in, is the weight of the first branch, n is the number of defects, k is the adjustment factor, and e is the base of the natural logarithm;

[0101] Formula (2);

[0102] in, is the weight of the second branch.

[0103] In this embodiment, the specific value of k can generally be 2. When the number of defects is large (i.e., the complexity of mixed defects is high), the weight of the second branch is automatically increased, strengthening the influence of positioning information on classification. This enables the model to more accurately identify and locate defects when processing complex mixed defects, thereby significantly improving detection accuracy.

[0104] In one embodiment of this specification, a joint loss function is used to optimize the fusion result to obtain a defect detection result, including the following formula (3):

[0105] (3);

[0106] in, Loss is the joint loss function; is the classification cross entropy loss; is the GIoU positioning loss; is the mean square error loss of the number of defects; for The coefficient of for The coefficient of .

[0107] In this embodiment, It is the loss function for the defect type prediction task, which measures the difference between the defect type predicted by the model and the actual defect type. It is a loss function for the defect bounding box prediction task, which measures the difference between the predicted bounding box and the true bounding box. The GIoU loss considers the minimum enclosing rectangle of the predicted box and the true box, and improves the positioning accuracy of the overlapping area. It is the loss function for the defect number prediction task, which measures the square error between the number of defects predicted by the model and the actual number of defects. The above parameters work together in the model training process to optimize the defect detection performance of the model by minimizing the joint loss function. and As weight coefficients, they control the contribution ratio of positioning loss and quantity loss in the total loss respectively, in order to achieve a balance between different tasks. Greater than ,For example, The value is 0.5, The value is 0.3.

[0108] In one embodiment of this specification, the wafer mixed defect detection method based on multimodal deep learning further includes:

[0109] Generate an analysis report based on the pre-established mapping relationship between defect detection results and process parameters; the analysis report includes defect cause analysis, tuning suggestions and expected results.

[0110] In this embodiment, a mapping relationship is a database or model that links defect detection results with process parameters. This relationship is established by collecting a large amount of historical data and analyzing the relationship between detection results for different defect types, locations, and quantities, and the corresponding process parameters. For example, if a certain defect type (such as edge ring defects) frequently occurs under specific process parameter settings (such as excessive etching pressure), this association is recorded in the mapping relationship.

[0111] Through this mapping, analysis reports can pinpoint specific process parameter issues that led to defects. For example, if a large number of edge ring defects are detected at the wafer edge, the report will analyze whether this is due to excessive edge pressure during the etching process. Another example is the presence of both ring defects and scratches, which could be linked to "excessive wafer transfer robot vibration." By quickly pinpointing the cause of defects and providing optimization recommendations, production downtime and debugging time caused by defects can be significantly reduced.

[0112] During cause analysis, a decision tree algorithm matches knowledge graph rules based on defect type, coordinate distribution, and quantity, outputting the top three possible causes (with a confidence level ≥ 80%). For example, "Abnormal etching pressure (92% confidence) > Uneven photoresist edge coating (85% confidence)." These optimization recommendations can be fed back to the production control system via an API interface, enabling automatic calibration of etching machine parameters.

[0113] In one embodiment of the present specification, before acquiring the multimodal image of the wafer, the following steps may be further included:

[0114] Plasma treatment is used to clean and modify the wafer surface, so that a uniform nano-scale texture structure is formed on the wafer surface.

[0115] In this embodiment, plasma is an ionized gas composed of electrons, ions, and neutral particles, possessing high energy and activity. Cleaning can remove organic contaminants, particulate matter, and other impurities from the wafer surface, reducing their interference with defect detection. Modification can create a uniform nanoscale texture on the wafer surface. This texture improves the optical properties of the wafer surface and enhances defect contrast during imaging, thus facilitating subsequent defect detection.

[0116] In some other embodiments of this specification, the following may also be included:

[0117] During the plasma treatment process, optical emission spectroscopy (OES) and ellipsometer are used for in-situ monitoring, and a fuzzy controller is used to provide real-time feedback and dynamic adjustment of plasma treatment parameters.

[0118] After the plasma treatment is completed, the wafer surface can also be post-processed, which may include:

[0119] A fluorine-containing silicon oxide passivation layer is formed on the wafer surface using chemical vapor deposition (CVD) technology to enhance the optical properties and chemical stability of the wafer surface;

[0120] The nanoscale texture structure on the wafer surface is modified by low-temperature plasma polymerization. By introducing monomer gas containing specific organic groups (such as monomers containing amino groups (such as 3-aminopropyltriethoxysilane, APTES)), a polymer film is formed in the texture structure to change the surface energy and chemical properties of the texture structure.

[0121] In this embodiment, the initial parameters of the plasma treatment, such as gas flow, power, pressure, etc., are set. The chemical reaction process in the plasma is monitored by OES, and the film thickness and roughness on the wafer surface are measured by ellipsometer. The monitoring data of the OES and ellipsometer are transmitted to the fuzzy controller in real time. The fuzzy controller calculates new control parameters according to the monitoring data and preset fuzzy rules, and adjusts the plasma treatment parameters in real time. The temperature range of low-temperature plasma polymerization is generally between room temperature and 100°C, and the specific temperature selection depends on the activity of the monomer gas, the performance requirements of the polymer film and the heat resistance of the substrate material. Through experimental verification, the optimal temperature conditions can be determined to achieve the ideal polymerization effect.

[0122] In one embodiment of this specification, a wafer mixed defect detection method based on multimodal deep learning further includes:

[0123] Based on the defect detection results, the defects on the wafer are marked using quantum dot fluorescent marking technology.

[0124] In this embodiment, quantum dots are combined with defect locations to emit fluorescence when excited by light of a specific wavelength, thereby clearly displaying the location and range of the defect under a microscope or other imaging equipment.

[0125] In some other embodiments of this specification, marking defects on a wafer using quantum dot fluorescent marking technology based on defect detection results may include:

[0126] Generate defect location map: Generate a defect location map containing defect location information based on the defect detection results;

[0127] Select quantum dots: Based on the defect type and detection requirements, select quantum dots with specific fluorescence properties. The fluorescence wavelength of the quantum dots corresponds to the defect type and has high quantum yield and good photostability.

[0128] Preparation of quantum dot solution: The selected quantum dots are dispersed in a suitable solvent to prepare a uniform quantum dot solution. The solvent is non-corrosive to the wafer surface and does not affect the fluorescence properties of the quantum dots.

[0129] Precise positioning and marking: Using micro-nano printing technology, the quantum dot solution is precisely printed on the defect location to form a fluorescent marking point. Micro-nano printing technology can achieve sub-micron positioning accuracy, ensuring high-precision alignment of the marking point and the defect location;

[0130] Fluorescence imaging verification: The marked wafer is imaged using a fluorescence microscope to verify the position and fluorescence intensity of the fluorescent marking points to ensure the accuracy and reliability of the marking.

[0131] In this embodiment, when selecting quantum dots, in addition to considering their fluorescence wavelength and quantum yield, the quantum dots can also be surface-modified. For example, by introducing specific ligands (such as polyethylene glycol ligands), the adhesion and stability of the quantum dots on the wafer surface can be improved, preventing the label from falling off during subsequent wafer processing. At the same time, surface modification can improve the biocompatibility and chemical stability of quantum dots, making them more suitable for use in different wafer materials and process environments. When preparing quantum dot solutions, specific enhancers (such as fluorescence-enhancing nanoparticles) can be added to improve the fluorescence intensity and stability of the quantum dots. For example, by conjugating or physically mixing, fluorescence-enhancing nanoparticles can be combined with quantum dots to significantly increase the fluorescence intensity of the quantum dots under excitation light and extend their fluorescence lifetime, thereby more clearly displaying the location and range of defects under a microscope or other imaging equipment, thereby improving the reliability of the labeling.

[0132] The present invention is described below through some specific examples.

[0133] Example 1: Hardware system integration:

[0134] Equipment: Modify the NOVA-2000-G glass wafer inspection equipment and add a differential interference lens and backside noise suppression module;

[0135] Dataset: MixedWM38 (38 defects) and WM-811K (172,950 labeled samples), split into training / validation / test sets at an 8:1:1 ratio.

[0136] Training process:

[0137] The trimodal images are data-enhanced (GAN generates rare defect samples, such as edge chipping samples, which increases by 3 times);

[0138] The two-branch network was trained on an NVIDIA A100 GPU with a batch size of 32 and a learning rate of 0.001 for 100 epochs.

[0139] Performance indicators: Hybrid defect detection accuracy is 99.2%, and positioning accuracy (IOU) is 95.3%, an 18.5% improvement over the existing Mask R-CNN.

[0140] Example 2, process optimization verification:

[0141] Defect case: It was detected that the edge ring defect (ER) ratio of a batch of wafers exceeded the threshold;

[0142] Cause Analysis: The process optimization engine matched the defect distribution (concentrated in edge areas) with historical data and identified the problem as "uneven photoresist coating thickness at the edge."

[0143] Adjustment effect: After adjusting the edge pressure parameters of the coater, the ER defect rate dropped from 8% to 1.2%, and the yield increased by 6.5%.

[0144] In summary, this invention effectively addresses the problem of single-modality inspection's insufficient ability to identify mixed defects. The synergistic effect of brightfield, darkfield, and differential interference imaging, combined with a dynamic weighted fusion algorithm, increases the detection accuracy of complex scenarios such as edge ring defects and random defects from below 85% with existing technologies to 99.2%, reducing missed detection rates by over 70%, significantly improving the reliability of defect detection in semiconductor manufacturing. To address the challenge of class imbalance, GAN-based data augmentation technology increases the sample size of rare defects by more than three times. Combined with manifold regularization to maintain data distribution consistency, it reduces the classification error rate of small sample defects (such as chipping and near-fill defects) by 60%, significantly enhancing model generalization. Furthermore, a knowledge-driven process optimization engine reduces defect cause tracing time from four hours to less than five minutes by analyzing the correlation between defect coordinates, types, and historical process parameters. It also automatically generates process adjustment recommendations, achieving a closed-loop feedback loop from inspection results to production optimization. This is expected to increase wafer yield by 5%-10%, effectively reducing resource waste and time costs in the manufacturing process.

[0145] Based on the same general inventive concept, the present invention also protects a wafer hybrid defect detection system based on multimodal deep learning, such as Figure 2 As shown, Figure 2 Schematic diagram of the structure of a wafer hybrid defect detection system based on multimodal deep learning provided by an embodiment of the present invention. The following describes the wafer hybrid defect detection system based on multimodal deep learning provided by the present invention. The wafer hybrid defect detection system based on multimodal deep learning described below and the wafer hybrid defect detection method based on multimodal deep learning described above can be referenced to each other.

[0146] The wafer hybrid defect detection system based on multimodal deep learning includes an image acquisition module 201, a preprocessing module 202, a channel stitching module 203, a defect analysis module 204 and a defect determination module 205.

[0147] The image acquisition module 201 is used to acquire multimodal images of the wafer; wherein the multimodal images include bright field images, dark field images and differential interference contrast images;

[0148] The pre-processing module 202 is used to perform normalization processing on the multimodal image;

[0149] The channel stitching module 203 is used to perform channel stitching on the normalized multimodal image to obtain a multi-channel input image;

[0150] The defect analysis module 204 is configured to detect the defect type, defect quantity, defect bounding box coordinates, a first probability distribution of the defect type, and a second probability distribution of the defect quantity of the multi-channel input image based on a dual-branch deep learning network;

[0151] The defect determination module 205 is configured to perform weighted fusion on the first probability distribution and the second probability distribution to obtain a defect detection result.

[0152] Figure 3 It is a structural diagram of an electronic device provided by an embodiment of the present invention.

[0153] like Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the wafer hybrid defect detection method based on multimodal deep learning.

[0154] Furthermore, the logic instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0155] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the wafer mixed defect detection method based on multimodal deep learning provided by the above methods.

[0156] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the wafer mixed defect detection method based on multimodal deep learning provided by the above-mentioned methods.

[0157] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0158] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A wafer mixed defect detection method based on multimodal deep learning, characterized in that: include: Acquire a multimodal image of the wafer; wherein the multimodal image includes a bright field image, a dark field image, and a differential interference contrast image; Normalizing the multimodal image; Perform channel splicing on the normalized multimodal image to obtain a multi-channel input image; Based on a dual-branch deep learning network, detect the defect type, defect quantity, defect bounding box coordinates, a first probability distribution of the defect type, and a second probability distribution of the defect quantity of the multi-channel input image; Performing weighted fusion on the first probability distribution and the second probability distribution to obtain a defect detection result; The dual-branch deep learning network includes a first branch and a second branch. The dual-branch deep learning network is used to detect the defect type, defect quantity, defect bounding box coordinates, a first probability distribution of the defect type, and a second probability distribution of the defect quantity of the multi-channel input image, including: Inputting the multi-channel input image into the lightweight ResNet network of the first branch, extracting texture features and geometric features of the defect, and generating a feature vector; Performing global average pooling on the feature vector to obtain a dimension-reduced feature vector; Input the reduced dimension feature vector into a fully connected layer, and output the defect type and the corresponding first probability distribution through a Sigmoid activation function; Inputting the multi-channel input image into the basic convolution of the second branch to extract initial features of the multi-channel input image; Based on the initial features, output the number of defects and the corresponding second probability distribution; reshape the second probability distribution of the number of defects into a feature map to characterize the spatial distribution of the number of defects; The initial multimodal image is input into the second branch, and the spatial features of the multimodal image are extracted through multi-layer convolution operations; Splicing the spatial features with the feature map to obtain a feature splicing map; Detecting the feature mosaic image to obtain defect bounding box coordinates; The weighted fusion of the first probability distribution and the second probability distribution to obtain a defect detection result includes: Dynamically adjust the weights of the first and second branches according to the number of defects; fusing the first probability distribution of the defect type and the second probability distribution of the defect quantity according to the weights of the first branch and the second branch to obtain a fusion result; Use the joint loss function to optimize the fusion results and obtain the defect detection results; The combined loss function is used to optimize the fusion result to obtain the defect detection result, including: ； in, Loss is the joint loss function, is the classification cross entropy loss, is the GIoU positioning loss, is the mean square error loss of the number of defects, for The coefficient of for The coefficient of .

2. The wafer hybrid defect detection method based on multimodal deep learning according to claim 1, characterized in that: The acquiring of a multimodal image of the wafer includes: illuminating the surface of the wafer with direct light to obtain the bright field image; Oblique light is used to illuminate the surface of the wafer to obtain the dark field image; The differential interference contrast image is obtained by illuminating the surface of the wafer with polarized light.

3. The wafer hybrid defect detection method based on multimodal deep learning according to claim 1, characterized in that: The dynamically adjusting the weights of the first branch and the second branch according to the number of defects includes: ； in, is the weight of the first branch, n is the number of defects, k is the adjustment factor, and e is the base of the natural logarithm; ； in, is the weight of the second branch.

4. The wafer hybrid defect detection method based on multimodal deep learning according to claim 1, characterized in that: Also includes: Generate an analysis report based on the pre-established mapping relationship between defect detection results and process parameters; wherein the analysis report includes defect cause analysis, tuning suggestions and expected results.

5. The wafer mixed defect detection method based on multimodal deep learning according to claim 1, characterized in that: Before acquiring multimodal images of the wafer, it also includes: Plasma treatment is used to clean and modify the wafer surface, so that a uniform nano-scale texture structure is formed on the wafer surface.

6. The wafer hybrid defect detection method based on multimodal deep learning according to claim 1, characterized in that: Also includes: According to the defect detection results, the defects on the wafer are marked using quantum dot fluorescent marking technology.

7. A wafer hybrid defect detection system based on multimodal deep learning, characterized in that: include: An image acquisition module, configured to acquire a multimodal image of the wafer; wherein the multimodal image includes a bright field image, a dark field image, and a differential interference contrast image; A preprocessing module, configured to perform normalization processing on the multimodal image; The channel stitching module is used to stitch the normalized multimodal images into channels to obtain a multi-channel input image; a defect analysis module, configured to detect the defect type, the number of defects, the coordinates of the defect bounding box, the first probability distribution of the defect type, and the second probability distribution of the number of defects of the multi-channel input image based on a dual-branch deep learning network; a defect determination module, configured to perform weighted fusion of the first probability distribution and the second probability distribution to obtain a defect detection result; The dual-branch deep learning network includes a first branch and a second branch. Based on the dual-branch deep learning network, the defect type, the number of defects, the coordinates of the defect bounding box, the first probability distribution of the defect type, and the second probability distribution of the number of defects of the multi-channel input image are detected, including: Inputting the multi-channel input image into the lightweight ResNet network of the first branch, extracting texture features and geometric features of the defect, and generating a feature vector; Performing global average pooling on the feature vector to obtain a dimension-reduced feature vector; Input the reduced dimension feature vector into a fully connected layer, and output the defect type and the corresponding first probability distribution through a Sigmoid activation function; Inputting the multi-channel input image into the basic convolution of the second branch to extract initial features of the multi-channel input image; Based on the initial features, output the number of defects and the corresponding second probability distribution; reshape the second probability distribution of the number of defects into a feature map to characterize the spatial distribution of the number of defects; The initial multimodal image is input into the second branch, and the spatial features of the multimodal image are extracted through multi-layer convolution operations; Splicing the spatial features with the feature map to obtain a feature splicing map; Detecting the feature mosaic image to obtain defect bounding box coordinates; The weighted fusion of the first probability distribution and the second probability distribution to obtain a defect detection result includes: Dynamically adjust the weights of the first and second branches according to the number of defects; fusing the first probability distribution of the defect type and the second probability distribution of the defect quantity according to the weights of the first branch and the second branch to obtain a fusion result; Use the joint loss function to optimize the fusion results and obtain the defect detection results; The combined loss function is used to optimize the fusion result to obtain the defect detection result, including: ； in, Loss is the joint loss function, is the classification cross entropy loss, is the GIoU positioning loss, is the mean square error loss of the number of defects, for The coefficient of for The coefficient of .

Images