Chip Defect Detection Method Based on Multi-Stage Feature Fusion and Dynamic Optimization

Abstract

This invention discloses a chip defect detection method based on multi-stage feature fusion and dynamic optimization. It improves the speed and capability of defect detection by designing a defect localization model based on intelligent initialization to narrow down the defect location; it captures subtle defects using a high-quality reconstruction model based on a multi-angle projection-oriented 3D microstructure reconstruction algorithm, effectively avoiding missed detections; and it enhances the stability of defect detection by using a dynamic optimization model combined with microbuffering, enabling the chip defect detection system to exhibit high accuracy even when facing unknown defects. Furthermore, it reduces false positives through data preprocessing and data fusion of various models, and improves image quality through denoising and image enhancement techniques. This method fully extracts rich internal and external information of the product without damaging precision components such as chips, improves the generalization ability of the detection algorithm, and effectively identifies and detects various complex defects.

Description

Technical Field

[0001] This invention belongs to the field of industrial defect detection technology, and relates to a chip defect detection method based on multi-stage feature fusion and dynamic optimization. Background Technology

[0002] Traditional industrial inspection processes typically begin by establishing clear defect detection standards based on the product's characteristics. Next, operators are trained on product inspection and equipment operation. Finally, operators use the naked eye or relevant inspection instruments to check the product's surface or shape for obvious defects. After inspection, operators record and analyze the results, reporting any defects to the relevant departments so they can understand the product quality situation and implement improvements.

[0003] However, with the rapid development of global science and technology, various high-tech products are emerging in an endless stream, and the internal structures of products are becoming increasingly complex. Under these circumstances, traditional industrial inspection is far from meeting the needs of defect detection for various products, and some automated inspection equipment has emerged, such as optical and ultrasonic equipment, which are widely used in defect detection. However, these inspection devices often only provide 2D images and are difficult to reflect the 3D structural information of products. More and more companies need a method that can both automate defect detection and non-destructively reflect the 3D structural information of precision products to improve the efficiency of factory defect detection.

[0004] While significant progress has been made in defect detection technology for semiconductor devices such as chips, very little of this research involves combining X-ray microscopy with unsupervised defect localization algorithms, high-precision 3D microstructure reconstruction algorithms, and incremental learning dynamic optimization algorithms. Current defect detection technologies struggle to address issues such as uneven sample distribution, the inability of 2D slicing techniques to comprehensively capture all key information, and the limited types of defects available for model training, which are prevalent in industrial CT inspection. Furthermore, they are unable to effectively identify and detect various complex defects, indicating that current defect detection technologies fall far short of meeting the needs of inspection personnel.

[0005] Looking at the research and development of defect detection technology for semiconductor devices such as chips around the world, we can see that, regardless of the country, the development and application of chip defect detection technology combining X-ray microscopy scanning technology with unsupervised learning defect localization algorithms, high-precision 3D microstructure reconstruction algorithms, and incremental learning dynamic optimization algorithms are all in the initial exploratory stage. Overall, it is not yet mature and requires extensive research to gradually improve this technology. With the recent rapid development of artificial intelligence, many new technologies have emerged, giving rise to numerous small and precise high-tech products and endowing many older products with new functions. As product types increase and internal structures become more complex, traditional industrial X-ray inspection technology is proving insufficient for defect detection of precision components. Under these circumstances, various high-tech factories urgently need a chip defect detection method that combines unsupervised defect localization, high-quality 3D reconstruction, and incremental learning optimization algorithms. Summary of the Invention

[0006] To address the aforementioned technical problems in existing technologies, this invention proposes a chip defect detection method based on multi-stage feature fusion and dynamic optimization. This method combines X-ray microscopy scanning technology with an unsupervised learning defect localization algorithm, a high-precision three-dimensional microstructure reconstruction algorithm, and an incremental learning dynamic optimization algorithm. This approach fully leverages the rich internal and external information of the product without damaging precision components such as chips, improves the generalization ability of the detection algorithm, and effectively identifies and detects various complex defects. The specific technical solution is as follows:

[0007] A chip defect detection method based on multi-stage feature fusion and dynamic optimization is proposed, which includes a defect localization model, a high-quality reconstruction model, and a dynamic optimization model, comprising:

[0008] The defect localization model is used to perform data preprocessing, image reconstruction, feature extraction and fusion operations on the defect-free image of the chip to generate a defect heatmap and binarize it to obtain the defect localization result.

[0009] The normalized projection data, defect heatmap, and defect location results of the chip are input into the high-quality reconstruction model. The high-quality reconstruction model performs cropping projection on the defect location results and normalized projection data, and then performs reconstruction and restoration processing to obtain local defect reconstruction data and high-quality three-dimensional volume data. Then, the high-quality three-dimensional volume data is sliced ​​at depth to obtain high-precision two-dimensional slices. The defect heatmap and local defect reconstruction data are compared to correct the defect heatmap.

[0010] Using a dynamic optimization model, defect heatmaps and high-precision two-dimensional slices are combined into defect samples and stored in a buffer. New defect categories are learned and identified to update the defect samples in the buffer.

[0011] Furthermore, 3D X-ray microscopy was used to scan the defect-free chip sample (i.e., the normal chip) from multiple angles to obtain defect-free images. Then through All defect-free image data are normalized to [0,1], where I nor This represents the original, defect-free image. Let μ represent an m×n dimensional real matrix; μ represents the mean of the defect-free image data, and σ represents the variance of the defect-free image data.

[0012] Furthermore, the defect localization model includes a progressive reconstruction model. For the normalized defect-free image, a random number method is used to determine whether to generate a pseudo-defect image. An adaptive sampling mechanism is used to control the ratio of defect-free images and pseudo-defect images in the input data, and the pseudo-defect image is used as the input image data I of the progressive reconstruction model. input Model training is performed; the pseudo-defect image is generated using a randomly generated defect masking technique on a defect-free image I. nor Generate by performing circular occlusion, linear occlusion, and localized erosion operations.

[0013] Furthermore, the encoder in the progressive reconstruction model obtains the input image data I input Extracting multi-scale features f k =f enconder,k (I input In the progressive reconstruction model, the decoder processes the obtained multi-scale features through operations such as transposed convolution and upsampling, ultimately obtaining the reconstructed image I. r =f deconder (f k ), f = {f1, f2, ..., f K} represents the multi-scale feature representation formed by extracting features layer by layer; f enconder,k (●) indicates the encoder; f deconder (·) represents the decoder, which enables the reconstruction of image I. r As close as possible to a defect-free image I nor The progressive reconstruction model uses a pixel-level error loss function to calculate the reconstructed image I. r and real, defect-free images I nor The pixel differences between the two images are evaluated using a structural similarity loss function to assess the reconstructed image I. r and real, defect-free images I nor Based on the structural similarity between them, gradient loss is used to ensure the integrity of edge information in the reconstructed image.

[0014] Furthermore, the defect localization model also includes a detection model, which takes the input image I... input Reconstructing Image Ir Residual image R diff and multi-scale feature maps from intermediate layers of the progressive reconstruction model network After being adjusted to the same size, the images are stitched together and input into the detection model. The residual image R diff Through R diff =|I input -I r The encoder in the detection model uses a convolutional neural network f. k =f encoder,k (I final This method extracts defect features at different scales, and uses an attention mechanism to weightedly combine these features to obtain the final comprehensive features. Among them, f k Represents the multi-scale feature map of the k-th layer; w k The attention weights of the k-th layer are represented by S; W represents the linear transformation matrix used for feature weighting; the decoder in the detection model uses a unified perceptron parsing network to decode and obtain the defect heatmap S. score =f discriminative (f fused Then, the defect heatmap is binarized to obtain the final defect location result. The defect area is located based on the defect location results, where M pred (i,j) is a binary defect mask, where 1 represents a defect region and 0 represents a normal region, and T is the defect discrimination threshold.

[0015] Furthermore, set the reconstruction threshold T. θ The defect heatmap S output by the detection model score The reconstruction probability score S is obtained by normalization using the Softmax function. per =Softmax(S score Then, the reconstructed probability score is processed by max pooling to obtain the anomaly score, i.e., the defect score θ = Globalmaxpool(Avg.pool(S per If θ > T θ Then, reconstruction is performed again, requiring the reconstructed image I to be... r Input into the progressive reconstruction model until θ≤T θ Only then can the reconstructed image I be represented. r With defect-free image I nor Similarly, the image reconstruction is complete, in which, Let m represent the number of rows in the image, n represent the number of columns in the image, and N represent the number of columns in the image. p This indicates the number of projected images, i.e., the different angles during X-ray scanning.

[0016] Furthermore, the projected images obtained after a comprehensive scan of the chip sample using 3DX X-ray microscopy are normalized to obtain normalized projection data. Then, filtered back projection is used to generate high-quality 3D reference volume data. Where μ p and σ p These are the mean and standard deviation of the projected data; P norm (θ,t) represents the normalized projection data, where θ represents the projection angle and t represents the horizontal coordinate of the projected image; g filter (P norm (θ,t)) represents the result of filtering the projection data; ω(θ,t) represents the weighting coefficient, which is used to balance the contribution of projection data from different angles to image reconstruction;

[0017] The centroids of the normalized projection data and the high-quality 3D reference volume data are calculated separately, and then a rigid transformation is performed to obtain the projection data and volume data, as well as the data pairs after registration.

[0018] Furthermore, the high-quality reconstruction model takes the registered data pairs, defect heatmap, and defect localization results as input. It extracts multi-scale features through an encoder, utilizes multi-layer convolution operations on edge and texture features in the two-dimensional image, and then uses a decoder to progressively upsample and employ an attention mechanism to recover the three-dimensional volume data, resulting in local defect reconstruction data and high-quality three-dimensional volume data. Then, a structural similarity calculation method is used to compare the defect heatmap region and the local defect reconstruction data. A false detection threshold is set; if the structural similarity value is lower than the false detection threshold, it indicates that the defect localization model has produced a false detection. The falsely detected region is then removed from the defect localization results to obtain a corrected localization result. Finally, the corrected localization result is remapped onto the defect heatmap to generate a corrected defect heatmap.

[0019] Furthermore, the dynamic optimization model stores the corrected defect heatmap and high-precision two-dimensional slices in a buffer. At the same time, the defect heatmap and high-precision two-dimensional slices are combined to obtain the defect region, and the initial sample set is constructed through the defect region.

[0020] Then, the initial samples are extracted and cut into small pieces and stored in the buffer. The sample features of each small piece are calculated, and representative samples are selected based on the sample features. Finally, the initial buffer is combined with the representative samples so that the buffer stores the most representative samples.

[0021] Next, the dynamic optimization model extracts features from the high-precision two-dimensional slices, and obtains the high-dimensional feature vector of the defect sample. By combining the high-dimensional feature vector of the defect with the defect heatmap, feature enhancement is performed. Then, the enhanced high-dimensional feature vector is used to calculate the path integral, extract features, and classify defects.

[0022] If a new category of defect appears, cluster the sample features of the new category of defect, select representative samples, add the new category samples to the buffer, update the buffer, and if the buffer overflows, delete the oldest category samples to maintain the optimal sample set.

[0023] Furthermore, during the training of the dynamic optimization model, the path integral method is used to record the trajectory of each parameter change. The importance of each parameter is calculated through the parameter change trajectory. Cross-entropy loss is used to learn new types of defects, and regularization loss is used to protect old types of defects.

[0024] The method of this invention solves the problems of sample imbalance, limited types of defects available for learning, difficulty in three-dimensional detection and analysis of object slices, and poor performance of detection systems when faced with new defects in traditional detection methods. It improves the generalization ability and detection accuracy of the model and enables industrial defect detection methods to cope with various new defect types through rapid training iteration. Attached Figure Description

[0025] Figure 1 This is a flowchart of the chip defect detection process based on multi-stage feature fusion and dynamic optimization according to an embodiment of the present invention.

[0026] Figure 2 This is a flowchart of the training phase of the intelligent initialization defect rapid screening and localization algorithm according to an embodiment of the present invention;

[0027] Figure 3 This is a flowchart of the inference stage of the intelligent initialization defect rapid screening and localization algorithm in this embodiment of the invention;

[0028] Figure 4 This is a flowchart of the training phase of the three-dimensional microstructure reconstruction algorithm for multi-angle projection according to an embodiment of the present invention.

[0029] Figure 5 This is a flowchart of the speculation stage of the three-dimensional microstructure reconstruction algorithm for multi-angle projection according to an embodiment of the present invention;

[0030] Figure 6 This is a flowchart of the dynamic model optimization algorithm combined with microbuffering in an embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0032] like Figure 1As shown, this invention discloses a chip defect detection method based on multi-stage feature fusion and dynamic optimization. It mainly includes an intelligent initialization-based rapid defect screening and localization algorithm, a three-dimensional microstructure reconstruction algorithm oriented towards multi-angle projection, and a dynamic model optimization algorithm combined with microbuffering. The intelligent initialization-based rapid defect screening and localization algorithm narrows down the location of defects, improving the speed and capability of defect detection. The three-dimensional microstructure reconstruction algorithm oriented towards multi-angle projection captures subtle defects, effectively avoiding missed detections. Data preprocessing and data fusion of various models reduce false detections. Denoising and image enhancement techniques improve image quality. The dynamic model optimization algorithm combined with microbuffering enhances model stability, enabling the chip defect detection system to exhibit high defect detection accuracy even when facing unknown defects.

[0033] like Figure 2 As shown, the intelligent initialization defect rapid screening and localization algorithm is designed with a defect localization model, which includes two sub-networks: a progressive reconstruction model and a detection model. Specifically, it includes the following steps:

[0034] Step 1: Use 3D X-ray microscopy to scan the defect-free chip sample (i.e., normal chip) from multiple angles to obtain a large number of defect-free two-dimensional images. Then through All defect-free image data are normalized to [0,1]. Where I nor A raw two-dimensional image representing a defect-free chip. Let μ represent an m×n dimensional real matrix; μ represents the mean of the defect-free image data, and σ represents the variance of the defect-free image data.

[0035] The second step involves using a random number method to determine whether to generate a pseudo-defect image for the normalized, defect-free image. An adaptive sampling mechanism is used to control the ratio of defect-free images to pseudo-defect images in the training data. The pseudo-defect images are then used as input data for training the progressive reconstruction model, which reduces the probability of false detection and improves the accuracy of subsequent defect localization. Furthermore, the progressive reconstruction model trained with pseudo-defect images, i.e., simulated abnormal images, can reconstruct high-quality two-dimensional images, thereby increasing the speed of defect detection by the detection model.

[0036] The pseudo-defect image I def =I nor ·(1-M mask )+N is the result of using a randomly generated defect mask technique on a defect-free image I. nor This process involves operations such as circular masking, linear masking, and localized etching to generate the chip. This method can address issues such as the very low probability of defects occurring during the manufacturing process of semiconductor devices like chips, and the often uneven distribution of defects.

[0037] When each image data is input into the defect localization model, a random number r ~ U(0,1) is generated. When t < T, the probability of a false defect image being used as input data is... If r > Ρ fake In this case, a defect-free image is used as the input data. If r ≤ P fake In this case, a pseudo-defect image is used as the input data. However, when t ≥ T, the probability of using a pseudo-defect image as input data is P. fake (t) = 1.0, and the input data consists entirely of pseudo-defect images. Where r represents a random number between 0 and 1; t represents the current training round number; T represents the total number of training rounds; and P... fake I represents the probability value of generating a fake defect image; def This represents a pseudo-defect image generated by masking non-defect areas; M mask ∈{0,1} m×n This represents a pseudo-defect region mask, where 1 indicates a defective region and 0 indicates a defect-free region; N ~ N(0,σ) 2 Let represent a Gaussian noise matrix with a mean of 0 and a variance of σ. 2 .

[0038] Step 3: The encoder in the progressive reconstruction model reconstructs the input image I... input Extracting multi-scale features f k =f enconder,k (I input The output feature maps from the second and seventh layers are used as part of the input to the detection model. The decoder in the progressive reconstruction model processes the obtained multi-scale features through operations such as transposed convolution and upsampling, ultimately obtaining the reconstructed image I. r =f deconder (f k Where, f = {f1, f2, ..., f} K} represents the multi-scale feature representation formed by extracting features layer by layer; f enconder,k (●) indicates the encoder; f deconder (●) represents the decoder, which enables the reconstruction of image I. r As close as possible to a defect-free image I nor .

[0039] Step 4: The progressive reconstruction model uses a pixel-level error loss function to calculate the reconstructed image I. r and real, defect-free images I nor The pixel-level error loss function expression for the pixel differences between them is:

[0040]

[0041] The structural similarity loss function is used to evaluate the reconstructed image I. r and real, defect-free images I nor The structural similarity between the components ensures that the texture information of the reconstructed image is not lost. The expression for the structural similarity loss function is:

[0042]

[0043] Gradient loss is used to ensure the integrity of edge information in the reconstructed image. The gradient loss expression is as follows:

[0044]

[0045] The final combined loss function is L rec =αL MSE +βL SSIM +γL grad , where α, β, γ are hyperparameters used to control the weights of the three loss functions mentioned above.

[0046] Step 5: Input image I input Reconstructing Image I r and multi-scale feature maps from intermediate layers of the progressive reconstruction model network The data is input into the detection model, and to enhance the contrast of the defect area, it is processed using R... diff =|I input -I r | Obtain the residual image R diff The residual image is also input into the detection model. Due to the multi-size feature map f k Having different resolutions h k ×w k It needs to be passed Adjust the resolution to the same size m×n. Finally, concatenate all processed data into a multi-channel tensor. The input is then fed into the detection model. Here, 3+K represents the original image channel number 3 plus the number of multi-scale feature layers K. This feature fusion method can alleviate the overfitting problem caused by overly obvious defects in artificially generated abnormal images.

[0047] Step 6: The encoder in the detection model uses a convolutional neural network f k =f encoder,k (I final This method extracts defect features at different scales to improve detection accuracy. An attention mechanism is then used to weight and combine features from different scales to enhance the detection capability for defects of varying sizes, resulting in a final comprehensive feature set. Among them, f k Represents the multi-scale feature map of the k-th layer; w krepresents the attention weight of the k-th layer; W represents the linear transformation matrix used for feature weighting.

[0048] Step 7: The decoder in the detection model uses Unified Perception Analysis Network Decoding is performed to obtain the defect heatmap S. score =f discriminative (f fused Then, the defect heatmap is binarized to obtain the final defect location result.

[0049] The defect area is located based on the defect location results. Where M... pred (i,j) is a binary defect mask, where 1 represents a defect region and 0 represents a normal region, and T is the defect discrimination threshold.

[0050] The detection model uses a focus loss function to compare the difference between the generated defect heatmap and the pseudo-defects generated on the defect-free image. The loss function value is used to adjust various parameters in the detection model to enhance the detection capability of small regions.

[0051] Step 8: After the defect localization model has been trained, deploy the model. For example... Figure 3 As shown, multi-angle projections obtained by real-time scanning using 3D X-ray microscopy technology. As input to the progressive reconstruction model, there is no need to generate pseudo-defect image data. Set the reconstruction threshold T. θ The defect heatmap S output by the detection model score The reconstruction probability score S is obtained by normalization using the Softmax function. per =Softmax(S score Then, the reconstructed probability score is processed by max pooling to obtain the anomaly score, i.e., the defect score θ = Global max pool(Avg.pool(S per If θ > T θ This indicates that the reconstructed image quality is too poor and needs to be reconstructed again. The reconstructed image I needs to be... r Input the data into the progressive reconstruction model and repeat steps two through seven above until θ ≤ T. θ Only then can the reconstructed image I be represented. r With defect-free image I nor Similarly, the image reconstruction is complete. Among them, Let m represent the number of rows in the image, n represent the number of columns in the image, and N represent the number of columns in the image. p This indicates the number of projected images, i.e., the different angles during X-ray scanning.

[0052] Step 9: End.

[0053] like Figure 4 and Figure 5As shown, the 3D microstructure reconstruction algorithm for multi-angle projection is designed with a high-quality reconstruction model, also known as a high-precision reconstruction model, and specifically includes the following steps:

[0054] Step 1: Scanning the projection image obtained by 3D X-ray microscopy. Perform a normalization operation to obtain normalized projection data. Using filtered back projection to generate high-quality 3D reference volume data Where μ p and σ p These are the mean and standard deviation of the projected data; P norm (θ,t) represents the normalized projection data, where θ represents the projection angle and t represents the horizontal coordinate of the projected image; g filter (P norm (θ,t)) represents the result of filtering the projection data; ω(θ,t) represents the weighting coefficient, which is used to balance the contribution of projection data from different angles to image reconstruction.

[0055] Step 2: To align the coordinates of the two, first calculate the centroid of the normalized projected data. centroid of high-quality 3D reference volume data Then perform a rigid transformation to obtain the projection data. Body data and the data pairs after registration (P) aligned V aligned ). Among them, P norm (i,j,k) represents the pixel values ​​of the projection data; V true (i,j,k) represents the pixel values ​​of the high-quality 3D reference data; d represents the depth layer number of the volume data; P aligned and V aligned These represent the registered projection data and volume data, respectively. This represents the rotation matrix, used to calculate the rotation relationship between the projected coordinate system and the filtered back-projection data coordinate system; In this context, t represents the translation vector, used to calculate how much the center of the generated volume data needs to be translated to project onto its data. This step is used to ensure P aligned and V aligned Spatial consistency.

[0056] Step 3: Register the data pairs (P) aligned V aligned ), Defect heat map S score and defect location results M pred The input is fed into a high-quality reconstruction model, which is then processed by an encoder.

[0057] f k =f enconder,k(P aligned Multi-scale features are extracted from k∈[1,K]. Multi-layer convolution is used to extract edge and texture features from the 2D image. Then, through the decoder, the 3D volume data V is meticulously recovered by progressive upsampling and attention mechanisms. r =f decoder (f1,f2,…,f K Among them, the high-quality reconstruction model uses mean squared error loss. and structural similarity loss To calculate the difference between the reconstructed volume data and the reference volume data, the total loss L is obtained. total =α·L MSE +β·L SSIM Then, the network weights are iteratively optimized using gradient descent. Here, α and β are weight parameters, and θ is the model parameter. This step uses the registered data to train the network model, enabling the network to learn how to generate high-quality three-dimensional volume data from a small amount of two-dimensional radial projection data.

[0058] Step 4: Slice the high-quality 3D reference volume data into 2D slices according to depth to obtain high-precision reference data slices F. k =Slice(V true ,k),k∈[1,d]. The defect location result M pred With high-precision reference data slice F k By combining the results, the actual structural slices can be obtained. Then use structural similarity calculation For the defect heatmap region S score and real structural slices Compare them.

[0059] Let the false detection threshold T be... S If the SSIM value is lower than the false detection threshold T S This indicates that the defect location model produced a false detection. Otherwise, the area marked on the defect heatmap is correct. Among these, μ represents a measure of structural similarity, with values ​​between 0 and 1. S ,μ F Defect heatmap S score and real structure slices The mean; σ S ,σ F Defect heatmap S score and real structure slices variance; σ SF C1 and C2 represent the covariance of the two variables; C1 and C2 are small constants to avoid division by zero. This step, through this comparison method, can not only reduce the probability of false detections but also improve the accuracy of model reconstruction.

[0060] Step 5: If false detections occur, remove the falsely detected areas from the defect location results to obtain the corrected location results. The corrected location results are then remapped onto the defect heatmap to generate a corrected defect heatmap. This improves detection accuracy and reduces the possibility of false detections. Specifically, if the structural similarity metric is less than the set false detection threshold, it is judged as a false detection, and the position value of that pixel in the localization result is changed to 0; otherwise, the original value remains unchanged.

[0061] Step 6: After the high-precision reconstruction model training is completed, the model will be deployed. It will no longer use data pairs for training; instead, it will use the actual scanned data P. input The detection results M of the detection model pred (i,j) is used to select the region of interest for projection data acquisition.

[0062] Step 7: Normalize the scanned data to obtain normalized data. The detection results from the detection model and the normalized data are clipped and projected to obtain clipped projection data P. crop =P input ·M pred , cut data P crop and normalized data The data were input into a pre-trained, high-quality reconstruction model for reconstruction, resulting in high-precision reconstructed data V of local defects. defect =f recon (P crop ) and complete reconstructed data

[0063] Step 8: High-quality reconstruction model utilizes high-resolution reconstruction data of the defect area V defect This is used to perform false positive detection. Finally, the reconstructed high-quality 3D volume data is sliced ​​according to depth to generate a high-quality 2D image. If no false positives occur, then S final =S score , otherwise

[0064] Step 9: End.

[0065] like Figure 6 As shown, the dynamic model optimization algorithm combined with microbuffering is designed with a dynamic optimization model, specifically including the following steps:

[0066] Step 1: In the initial stage, the defect heatmap S final High-precision 2D slices generated by high-quality reconstruction models Store in buffer The buffer is used to efficiently store newly detected anomalies and defect samples, effectively mitigating catastrophic forgetting and improving detection accuracy.

[0067] Then, the defect area is obtained by combining the defect heat map with high-precision two-dimensional slices. An initial sample set is constructed using defect regions, and initial samples are extracted first. The p×p blocks cut from the initial sample will be stored in buffer B. init ={F1,F2,…,F s Then calculate the sample features f. i =f enconder (F i Representative samples F are selected based on sample characteristics. rep =Cluster(f i ),|F rep |=k, and finally the initial buffer is combined with the representative sample to ensure that the buffer stores the most representative sample B. samples =B init ∪F rep Among them, B samples represents the reference buffer, s represents the buffer capacity, and m×n represents the two-dimensional resolution of a single sample; Representing two-dimensional slices containing only defective regions helps reduce the computational cost of the model and improve classification accuracy; F i Indicates the extracted defective sample; B init This represents the initial reference buffer, which stores defect samples of known categories; F rep This represents a set of representative samples, containing the k most representative samples.

[0068] Step 2: The dynamic optimization model extracts features from the high-precision two-dimensional slices, resulting in the high-dimensional feature vector of the defective samples. Feature enhancement is performed by combining the high-dimensional feature vector of the defect with the defect heatmap. aug =f k ·S final .

[0069] Step 3: The dynamic optimization model uses the enhanced high-dimensional feature vectors to calculate path integrals, extract features, and classify defects.

[0070] During the training of a dynamic optimization model, the path integral method is used to record the trajectory of changes in each parameter. The model calculates the importance of each parameter by analyzing the trajectory of parameter changes.

[0071] The model's total loss function is in Represents parameter θ k The trajectory of change in the nth round of training, g k (θ(t)) represents the gradient of the parameters, θ' k (t) indicates the direction of parameter update; Represents parameter θ k Importance of all past tasks ξ represents the change in parameters, and ξ represents the damping factor to prevent the denominator from being 0.

[0072] The model utilizes cross-entropy loss. To learn about new categories of defects, D n Let f(x) represent the dataset of the new category, and f(x) represent the model's predicted probability for sample x.

[0073] The model utilizes regularized loss This is to protect against defects in older categories and prevent catastrophic forgetting.

[0074] This step involves the dynamic optimization model batch processing new data stored in the buffer, using path integrals to calculate the importance of various parameters, and continuously optimizing the model's defect detection capabilities. When the model encounters unrecognized defect types, it can adjust weights by comparing the differences between the new defect types and existing defect types, enabling the dynamic optimization model to identify new defect types. Simultaneously, by reasonably combining newly added data with existing data, the model's ability to identify older defect types is not lost, effectively mitigating catastrophic forgetting.

[0075] Step 4: If a new category of defect emerges, cluster the sample features of the new category of defect and select representative samples F. rep =Cluster(f Cn Add new category samples to buffer B. samples =B samples ∪F rep Update the buffer; if the buffer overflows, delete the oldest class sample to maintain the optimal sample set.

[0076] Step 5: After the dynamic optimization model training is complete, the model is deployed. The model input is still the defect heatmap and high-precision 2D slices. Feature extraction is performed on the 2D slices, followed by defect classification, and the probability of the defect category is calculated. To determine which category the defect belongs to. Where P(C) i |F k ) represents sample C i Belongs to category F k The probability of f classify (F k C i ) represents the classification network for Ci The output score; c represents the number of defect categories that the current model has learned.

[0077] Step 6: The dynamic optimization model will select the category with the highest probability based on the calculated probabilities. The classification result is output. A class confidence threshold T is set. C If maxΡ(C i |F k ) < T C Then it is considered that sample F k If a defect might belong to a new category, its features, heatmap, and model weights are stored in a buffer for future model updates. If a new category is detected, the dynamic optimization model is optimized by calculating its loss function and adjusting parameter importance to allow the model to gradually adapt to the new category and improve long-term classification performance.

[0078] Step 7: Dynamic optimization model generates a comprehensive defect diagnostic report. Inspection personnel use this report to analyze defect causes and improve processes. The report includes not only the type, location, and severity of defects, but also an assessment of their impact on the performance of semiconductor devices such as chips. The report can also visualize defects in samples using 3D visualization, helping inspection personnel quickly identify problems and minimize losses.

[0079] Step 8: End.

[0080] In summary, previous model training relied on datasets with balanced positive and negative samples. However, defects are not frequently generated in actual industrial manufacturing processes, making it difficult to obtain sufficiently large and diverse datasets. This invention's intelligent initialization-based rapid defect screening and localization algorithm expands normal samples and simulated anomalous samples through data augmentation techniques. Using an adaptive sampling mechanism and a random number judgment method, it randomly selects either anomaly-free images (defect-free images) or simulated anomalous images (pseudo-defect images) as input data for the progressive reconstruction model, reducing the probability of false detections and improving the accuracy of subsequent defect localization. The progressive reconstruction model trained on simulated anomalous images can reconstruct high-quality two-dimensional images, improving the speed of defect detection. Since there is no one-to-one correspondence between the reconstructed image and its original image at the pixel level, pixel-by-pixel comparison-based anomaly identification methods often lead to higher false detection rates. To address this issue, a deep learning-based detection model is used to detect anomalous regions at the feature and semantic levels, thereby improving the accuracy of anomaly detection. Secondly, compared to real defects, defect regions in simulated anomalous images typically have clear edges and patterns significantly different from normal regions. This makes simulated anomalies stand out clearly in the image, potentially leading to overfitting during the training of the detection model. To address this issue, the detection model not only takes the original image, reconstructed image, and residual image as input, but also integrates intermediate layer feature information obtained during the reconstruction process. By fusing these images and intermediate layer feature contents as input data to the detection model, the tendency of the model to rely solely on clear boundaries to locate abnormal regions can be reduced, thus effectively mitigating the overfitting problem.

[0081] To address the limitations of traditional defect detection algorithms, such as reliance on layer selection and lack of 3D detection and analysis capabilities, this invention employs a novel 3D microstructure reconstruction algorithm oriented towards multi-angle projection. First, preprocessing operations, including denoising and artifact reduction, are performed on the 2D projection data obtained from a comprehensive scan using 3D X-ray microscopy, significantly improving the reconstruction quality of the high-precision reconstruction model. Then, the preprocessed data is input into the high-precision reconstruction model to obtain high-quality 3D volumetric data. During the training phase, the high-precision reconstruction model spatially registers the high-quality 3D volumetric data obtained through filtered back-projection with the 2D projection data. After training with the registration pair, the model's reconstruction capability is significantly enhanced. By studying high-precision local defect reconstruction images and overall reconstruction images, the defect detection system can easily detect anomalies such as bubbles, cracks, and weld voids. Secondly, the high-precision reconstruction model compares the defect area generated by the defect localization model with the reconstructed high-precision area, reducing the probability of false detections and improving the model's reconstruction accuracy. Finally, the high-precision reconstruction model stores complete 3D data, facilitating manual verification by inspection personnel and providing rich foundational data for subsequent automated inspection.

[0082] Current chip defect detection methods mostly rely on training with fixed datasets, resulting in limited defect types that can be learned and poor generalization ability in practical deployments, leading to difficulties in achieving optimal performance. This invention's dynamic optimization model, combined with a buffer, stores defect detection-related content such as abnormal region data and 2D data slices in the buffer. Model parameters are adjusted by batch processing the data in the buffer, improving the model's generalization ability. Secondly, traditional models often suffer from catastrophic forgetting during training. However, the algorithm combining the dynamic optimization model with a buffer effectively mitigates catastrophic forgetting and improves detection accuracy because the buffer efficiently stores newly detected abnormal data and defect samples, prioritizing representative abnormal data and defect samples. The trained dynamic optimization model significantly improves the ability to identify new defects, and even when dealing with previously unseen defect types, the dynamic optimization model maintains excellent performance.

[0083] In the field of defect detection for semiconductor devices such as chips, real-time detection and processing capabilities are crucial to semiconductor device production capacity. Only by promptly detecting defects in the scanned device can corresponding production line adjustments be made to prevent significant losses in the factory. This invention rapidly locates potential defect positions using an intelligent initialization-based defect screening and localization algorithm. Then, a 3D microstructure reconstruction algorithm oriented towards multi-angle projection performs a cropping operation on the potential defect positions, using the cropped area for high-precision reconstruction. The high-precision reconstruction results are compared with the cropped area to determine whether false detections have occurred, thus providing rapid feedback for chip production lines. Finally, a dynamic model optimization algorithm combined with microbuffering utilizes a storage area to store the most representative defect samples. Processing the most representative samples each time mitigates catastrophic forgetting, improves the real-time performance and detection capabilities of the chip defect detection system, and significantly reduces the system's data processing time.

[0084] Furthermore, in the field of defect detection for semiconductor devices such as chips, accuracy directly affects multiple aspects, including production line quality control, semiconductor device reliability, and cost control. However, in practical applications, many factors influence the accuracy of defect detection, primarily including missed detections, false detections, and noise interference. This invention improves the speed and capability of defect detection by using an intelligent initialization-based rapid defect screening and localization algorithm to narrow down the location of defects; it effectively avoids missed detections by using a three-dimensional microstructure reconstruction algorithm oriented towards multi-angle projection to capture subtle defects; it reduces false detections through data preprocessing and data fusion of various models; it improves image quality through denoising and image enhancement techniques; and it enhances model stability by using a dynamic model optimization algorithm combined with microbuffering, enabling the chip defect detection system to maintain high accuracy even when facing unknown defects. By comprehensively utilizing the above technologies, the chip defect detection system can reliably and stably perform chip defect detection tasks and maintain high accuracy even in complex and changing production environments.

[0085] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the implementation process of the present invention has been described in detail above, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A chip defect detection method based on multi-stage feature fusion and dynamic optimization, comprising a defect localization model, a high-quality reconstruction model, and a dynamic optimization model, characterized in that, include: The defect localization model is used to perform data preprocessing, image reconstruction, feature extraction and fusion operations on the defect-free image of the chip to generate a defect heatmap and binarize it to obtain the defect localization result. The normalized projection data, defect heatmap, and defect location results of the chip are input into the high-quality reconstruction model. The high-quality reconstruction model performs cropping projection on the defect location results and normalized projection data, and then performs reconstruction and restoration processing to obtain local defect reconstruction data and high-quality three-dimensional volume data. Then, the high-quality three-dimensional volume data is sliced ​​at depth to obtain high-precision two-dimensional slices. The defect heatmap and local defect reconstruction data are compared to correct the defect heatmap. Using a dynamic optimization model, defect heatmaps and high-precision two-dimensional slices are combined into defect samples and stored in a buffer. New defect categories are learned and identified to update the defect samples in the buffer. The projected images obtained after a full scan of the chip sample using 3DX X-ray microscopy are normalized to obtain normalized projection data. Then, filtered back projection is used to generate high-quality 3D reference volume data. ,in and These are the mean and standard deviation of the projected data; This represents the normalized projection data. Indicates the projection angle. Represents the horizontal coordinate of the projected image; This represents the result of filtering the projected data; This represents the weighting coefficient, used to balance the contribution of projection data from different angles to image reconstruction; The centroids of the normalized projection data and the high-quality 3D reference volume data are calculated separately, and then a rigid transformation is performed to obtain the projection data and volume data, as well as the data pairs after registration. The high-quality reconstruction model takes into account the registered data pairs, defect heatmaps, and defect localization results. It extracts multi-scale features through an encoder, utilizes multi-layer convolution operations on edge and texture features in the 2D image, and then uses a decoder to progressively upsample and employ an attention mechanism to recover the 3D volume data, resulting in local defect reconstruction data and high-quality 3D volume data. Then, a structural similarity calculation method is used to compare the defect heatmap region and the local defect reconstruction data. A false detection threshold is set; if the structural similarity value is lower than the false detection threshold, it indicates that the defect localization model has produced a false detection. The falsely detected region is then removed from the defect localization results, resulting in a corrected localization result. Finally, the corrected localization result is remapped onto the defect heatmap to generate a corrected defect heatmap. The dynamic optimization model stores the corrected defect heatmap and high-precision two-dimensional slices in a buffer. At the same time, the defect heatmap and high-precision two-dimensional slices are combined to obtain the defect region, and the initial sample set is constructed through the defect region. Then, the initial samples are extracted and cut into small pieces and stored in the buffer. The sample features of each small piece are calculated, and representative samples are selected based on the sample features. Finally, the initial buffer is combined with the representative samples so that the buffer stores the most representative samples. Next, the dynamic optimization model extracts features from the high-precision two-dimensional slices, and obtains the high-dimensional feature vector of the defect sample. By combining the high-dimensional feature vector of the defect with the defect heatmap, feature enhancement is performed. Then, the enhanced high-dimensional feature vector is used to calculate the path integral, extract features, and classify defects. If a new category of defect appears, cluster the sample features of the new category of defect, select representative samples, add the new category samples to the buffer, update the buffer, and if the buffer overflows, delete the oldest category samples to maintain the optimal sample set.

2. The chip defect detection method as described in claim 1, characterized in that, 3D X-ray microscopy was used to scan defect-free chip samples (i.e., normal chips) from multiple angles to obtain defect-free images. Then through Normalize all defect-free image data to ,in This represents the original, defect-free image. Represent a A 3D real matrix; This represents the mean of defect-free image data. This represents the variance of defect-free image data.

3. The chip defect detection method as described in claim 2, characterized in that, The defect localization model includes a progressive reconstruction model. For the normalized defect-free image, a random number method is used to determine whether to generate a pseudo-defect image. An adaptive sampling mechanism is used to control the ratio of defect-free images and pseudo-defect images in the input data, and the pseudo-defect images are used as the input image data for the progressive reconstruction model. Model training is performed; the pseudo-defect image is generated using a randomly generated defect masking technique on a defect-free image. Generate by performing circular occlusion, linear occlusion, and localized erosion operations.

4. The chip defect detection method as described in claim 3, characterized in that, The encoder in the progressive reconstruction model reads the input image data. Extracting multi-scale features In the progressive reconstruction model, the decoder processes the obtained multi-scale features through transposed convolution and upsampling operations to finally obtain the reconstructed image. , This represents the multi-scale feature representation formed by extracting features layer by layer. Indicates encoder; The decoder enables the reconstruction of the image. Near-defect-free images The progressive reconstruction model uses a pixel-level error loss function to calculate the reconstructed image. and real, flawless images The pixel differences between the images are evaluated using a structural similarity loss function to assess the reconstructed image. and real, flawless images Based on the structural similarity between them, gradient loss is used to ensure the integrity of edge information in the reconstructed image.

5. The chip defect detection method as described in claim 4, characterized in that, The defect localization model also includes a detection model, which takes the input image as an input. Reconstructing images residual image and multi-scale feature maps from intermediate layers of the progressive reconstruction model network After being adjusted to the same size, the residual images are stitched together and input into the detection model. pass It was found that the encoder in the detection model uses a convolutional neural network. To extract defect features at different scales, an attention mechanism is used to weight and combine the features at different scales to obtain the final comprehensive features. ,in, Indicates the first Multi-scale feature maps of layers; Indicates the first Attention weights of layers; The linear transformation matrix is ​​used for feature weighting; the decoder in the detection model uses a unified perceptron parsing network to decode and obtain the defect heatmap. Then, the defect heatmap is binarized to obtain the final defect location result. The defect area is located based on the defect location results. This is a binary defect mask, where 1 represents a defective area and 0 represents a normal area. It is the defect detection threshold.

6. The chip defect detection method as described in claim 5, characterized in that, Set the reconstruction threshold The defect heatmap output by the detection model The reconstruction probability score is obtained by normalization using the Softmax function. The reconstruction probability score is then subjected to max pooling to obtain the anomaly score, i.e., the defect score. ;like If so, reconstruction is performed again, requiring the reconstructed image to be... Input into the progressive reconstruction model until... Only then can it be considered a reconstructed image. With defect-free images Similarly, the image reconstruction is complete, in which, Represents a three-dimensional data tensor. Indicates the number of rows in the image. Indicates the number of columns in the image. This indicates the number of projected images, i.e., the different angles during X-ray scanning.

7. The chip defect detection method as described in claim 1, characterized in that, During the training of the dynamic optimization model, the path integral method is used to record the trajectory of the changes of each parameter. The importance of each parameter is calculated by the trajectory of the parameter changes. Cross-entropy loss is used to learn new types of defects, and regularization loss is used to protect old types of defects.

Images