Chip defect detection method based on multi-stage feature fusion and dynamic optimization

Abstract

The invention discloses a chip defect detection method based on multi-stage feature fusion and dynamic optimization, and the method comprises the steps: designing a defect positioning model based on intelligent initialization to reduce the position of a defect, and improving the speed and capability of defect detection; fine defects are captured through a high-quality reconstruction model based on a three-dimensional microstructure reconstruction algorithm oriented to multi-angle projection, and missing detection can be effectively avoided; the stability of defect detection is improved by using a dynamic optimization model combined with micro-buffering, so that the chip defect detection method system can show relatively high defect detection accuracy even in the face of unknown defects. False detection is reduced through data preprocessing and model result data fusion, and image quality is improved through denoising and image enhancement technologies. According to the method, rich information inside and outside a product is fully mined on the basis of not damaging precise elements such as a chip, the generalization ability of a detection algorithm is improved, and various complex defects are effectively recognized and detected.

Description

Technical Field

[0001] The present invention belongs to the technical field of industrial defect detection and relates to a chip defect detection method based on multi-stage feature fusion and dynamic optimization. Background Art

[0002] Traditional industrial inspection processes often begin by establishing clear defect detection standards based on product characteristics. Operators are then trained on product testing and equipment operation. Finally, operators visually inspect the product surface or shape for obvious defects, either visually or using relevant testing instruments. After inspecting the product, operators record and statistically analyze the results, reporting any defects to the relevant departments to ensure they understand product quality and implement improvements.

[0003] However, with the rapid development of global technology, a wide range of high-tech products are emerging, and their internal structures are becoming increasingly complex. In this context, traditional industrial inspection is no longer sufficient for defect detection in various products. Automated inspection equipment, such as optical and ultrasonic methods, has emerged and is widely used in defect detection. However, these inspection devices often only provide 2D images and are unable to reflect the product's 3D structure. More and more companies are seeking a method that can both automate defect detection and non-destructively capture the 3D structure of precision products to improve factory defect detection efficiency.

[0004] While research has yielded some promising results in defect detection for semiconductor devices like chips, few of these efforts integrate X-ray microscopy with unsupervised defect localization algorithms, high-precision 3D microstructure reconstruction algorithms, and incremental learning dynamic optimization algorithms. Current industrial CT inspections face challenges such as uneven sample distribution, the difficulty of 2D slicing technology in capturing all key information, and a limited number of defect types for model training. Existing defect detection technologies struggle to address these challenges and are unable to effectively identify and detect a wide range of complex defects. This demonstrates that existing defect detection technology falls far short of meeting the demands of inspectors in this area.

[0005] Looking at the research and development of chip and other semiconductor device defect detection technologies around the world, we find that, regardless of country, the development and application of chip defect detection technologies—combining X-ray microscopy with unsupervised learning defect localization algorithms, high-precision 3D microstructure reconstruction algorithms, and incremental learning dynamic optimization algorithms—are still in the initial exploratory stages. Overall, these technologies are still immature, requiring extensive research to gradually refine. The recent boom in artificial intelligence has ushered in numerous new technologies, created numerous small and precise high-tech products, and given existing products new functionality. With the increasing variety of products and their increasingly complex internal structures, traditional industrial X-ray inspection technology is struggling to detect defects in these precision components. In this context, high-tech factories are in urgent need of a chip defect detection method that combines unsupervised defect localization, high-quality 3D reconstruction, and incremental learning optimization algorithms. Summary of the Invention

[0006] To address the above-mentioned technical problems existing in the prior art, the present invention proposes a chip defect detection method based on multi-stage feature fusion and dynamic optimization. This method combines X-ray microscopy scanning technology with an unsupervised learning defect localization algorithm, a high-precision three-dimensional microstructure reconstruction algorithm, and an incremental learning dynamic optimization algorithm. This method fully mines the rich internal and external information of the product without damaging precision components such as the chip, improves the generalization ability of the detection algorithm, and effectively identifies and detects various complex defects. The specific technical solution is as follows:

[0007] A chip defect detection method based on multi-stage feature fusion and dynamic optimization is designed. It includes a defect localization model, a high-quality reconstruction model, and a dynamic optimization model, including:

[0008] The defect localization model is used to perform data preprocessing, image reconstruction, and feature extraction and fusion operations on the defect-free image of the chip to generate a defect heat map and binarize it to obtain the defect localization result;

[0009] The normalized projection data, defect heat map and defect location results of the chip are input into a high-quality reconstruction model. The high-quality reconstruction model crops and projects the defect location results and normalized projection data, and then performs reconstruction and recovery processing to obtain local defect reconstruction data and high-quality three-dimensional volume data. The high-quality three-dimensional volume data is then deep sliced to obtain high-precision two-dimensional slices. The defect heat map and the local defect reconstruction data are compared to correct the defect heat map.

[0010] Using a dynamic optimization model, defect heat maps and high-precision two-dimensional slices are combined into defect samples and stored in a buffer. New categories of defects that appear are learned and identified to update the defect samples in the buffer.

[0011] Furthermore, 3D X-ray microscopy technology is used to scan the defect-free chip sample (normal chip) at multiple angles to obtain a defect-free image. Then pass All defect-free image data are normalized to [0,1], where I nor represents the original defect-free image, represents an m×n dimensional real matrix; μ represents the mean of the defect-free image data, and σ represents the variance of the defect-free image data.

[0012] Furthermore, the defect location model includes a progressive reconstruction model. For the normalized defect-free image, a random number judgment method is used to determine whether to generate a pseudo-defect image. An adaptive sampling mechanism is used to control the ratio of the defect-free image and the pseudo-defect image in the input data, and the pseudo-defect image is used as the input image data I of the progressive reconstruction model. input The pseudo defect image is a random defect mask generated by the defect-free image I nor Perform circular occlusion, linear occlusion and local erosion operations to generate.

[0013] Furthermore, the encoder in the progressive reconstruction model is fed with input image data I input Extract multi-scale features f k =f enconder,k (I input ), the decoder in the progressive reconstruction model processes the obtained multi-scale features through operations such as transposed convolution and upsampling, and finally obtains the reconstructed image I r =f deconder (f k ), f={f1,f2,…,f K} represents the multi-scale feature representation formed by extracting features layer by layer; f enconder,k (●) indicates the encoder; f deconder (·) represents the decoder so that the reconstructed image I r As close to a defect-free image as possible nor ; Among them, the progressive reconstruction model uses a pixel-level error loss function to calculate the reconstructed image I r and the true defect-free image I nor The pixel difference between them is used to evaluate the reconstructed image I using the structural similarity loss function. r and the true defect-free image I nor The structural similarity between the two images is calculated, and the gradient loss is used to ensure the edge information of the reconstructed image is complete.

[0014] Furthermore, the defect location model also includes a detection model, which inputs the image I input , reconstructed image Ir , residual image R diff and multi-scale feature maps from the intermediate layers of the progressive reconstruction model network After being adjusted to the same size, the residual image R diff By R diff =|I input -I r |Get, the encoder in the detection model uses a convolutional neural network f k =f encoder,k (I final ) to extract defect features of different scales, and use the attention mechanism to weightedly combine features of different scales to obtain the final comprehensive features Among them, f k represents the multi-scale feature map of the kth layer; w k Represents the attention weight of the kth layer; W represents the linear transformation matrix, which is used for feature weighting; the decoder in the detection model uses a unified perceptual parsing network for decoding to obtain the defect heat map S score =f discriminative (f fused ), and then binarize the defect heat map to obtain the final defect location result Locate the defect area through the defect location results, where M pred (i, j) is the binary defect mask, 1 represents the defect area, 0 represents the normal area, and T is the defect discrimination threshold.

[0015] Furthermore, the reconstruction threshold T is set θ , the defect heat map S output by the detection model score Use the Softmax function to normalize and get the reconstruction probability score S per =Softmax(S score ), and then the reconstruction probability score is processed by the maximum pooling to obtain the abnormal score, that is, the defect score θ = Globalmaxpool(Avg.pool(S per )); if θ>T θ , then reconstruct again, and reconstruct the image I r Input into the progressive reconstruction model until θ≤T θ It represents the reconstructed image I r With the defect-free image I nor Similarly, image reconstruction is completed, where Represents a three-dimensional data tensor, m represents the number of rows of the image, n represents the number of columns of the image, N p Indicates the number of projection images, that is, different angles during X-ray scanning.

[0016] Furthermore, the projection image obtained after the chip sample is fully scanned by 3D X-ray microscope technology is normalized to obtain normalized projection data. Then use filtered back projection to generate high-quality 3D reference volume data where μ p and σ p is the mean and standard deviation of the projected data; P norm (θ, t) represents the normalized projection data, θ represents the projection angle, and t represents the horizontal coordinate of the projection image; g filter (P norm (θ, t)) represents the result of filtering the projection data; ω(θ, t) represents the weighting coefficient, which is used to balance the contribution of projection data at different angles to image reconstruction;

[0017] The centroid of the normalized projection data and the centroid of the high-quality three-dimensional reference volume data are calculated respectively, and then a rigid transformation is performed to obtain the projection data and volume data, as well as the data pair after the two are registered.

[0018] Furthermore, the high-quality reconstruction model inputs the registered data pairs, defect heat maps and defect localization results, extracts multi-scale features through the encoder, utilizes multi-layer convolution operations on the feature information about edges and textures in the two-dimensional image, and then gradually upsamples and uses the attention mechanism of the decoder to restore the three-dimensional volume data to obtain local defect reconstruction data and high-quality three-dimensional volume data; then a structural similarity calculation method is used to compare the defect heat map area and the local defect reconstruction data, and a false detection threshold is set. If the value of the structural similarity is lower than the false detection threshold, it means that the defect localization model has generated a false detection, and the falsely detected area is removed from the defect localization result to obtain a corrected localization result, and then the corrected localization result is remapped to the defect heat map to generate a corrected defect heat map.

[0019] Furthermore, the dynamic optimization model stores the corrected defect heat map and the high-precision two-dimensional slices in a buffer, and combines the defect heat map with the high-precision two-dimensional slices to obtain the defect area, and constructs an initial sample set based on the defect area;

[0020] Then, the initial sample is extracted and cut into small blocks and stored in the buffer. The sample features of each block are calculated, and representative samples are selected based on the sample features. Finally, the initial buffer is combined with the representative samples so that the most representative samples are stored in the buffer.

[0021] The dynamic optimization model then extracts features from high-precision two-dimensional slices to obtain high-dimensional feature vectors of the defect samples. This feature vector is then combined with the defect heat map to enhance the features. The enhanced high-dimensional feature vector is then used to calculate path integrals, extract features, and classify defects.

[0022] If a new category of defects appears, the sample features of the new category of defects are clustered, representative samples are selected, the new category samples are added to the buffer, and the buffer is updated. If the buffer overflows, the oldest category samples are deleted to maintain the optimal sample set.

[0023] Furthermore, during the dynamic optimization model training process, the path integral method is used to record the trajectory of each parameter change, and the importance of each parameter is calculated through the parameter change trajectory. The cross entropy loss is used to learn new category defects, and the regularization loss is used to protect old category defects.

[0024] The method of the present invention solves the problems of sample imbalance, limited defect types available for learning, difficulty in three-dimensional detection and analysis of object slices, and poor performance of the detection system in the face of new defects in traditional detection methods, thereby achieving the effect of improving the generalization ability and detection accuracy of the model, and through rapid training iteration, it can be used to cope with various new types of industrial defect detection methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a comprehensive flow chart of chip defect detection based on multi-stage feature fusion and dynamic optimization according to an embodiment of the present invention;

[0026] Figure 2 1 is a flow chart of the training phase of the intelligently initialized defect rapid screening and location algorithm according to an embodiment of the present invention;

[0027] Figure 3 This is a flow chart of the inference phase of the intelligently initialized defect rapid screening and location algorithm according to an embodiment of the present invention;

[0028] Figure 4 This is a flow chart of the training phase of a three-dimensional microstructure reconstruction algorithm for multi-angle projection according to an embodiment of the present invention;

[0029] Figure 5 is a flow chart of the inference phase of a three-dimensional microstructure reconstruction algorithm for multi-angle projection according to an embodiment of the present invention;

[0030] Figure 6 FIG. 4 is a flow chart of a dynamic model optimization algorithm combined with micro-buffering according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and technical effect of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0032] like Figure 1As shown, the present invention discloses a chip defect detection method based on multi-stage feature fusion and dynamic optimization, which mainly includes an intelligently initialized defect rapid screening and positioning algorithm, a three-dimensional microstructure reconstruction algorithm for multi-angle projection, and a dynamic model optimization algorithm combined with a micro-buffer. The intelligently initialized defect rapid screening and positioning algorithm is used to narrow the location of the defect, thereby improving the speed and capability of defect detection; the three-dimensional microstructure reconstruction algorithm for multi-angle projection is used to capture subtle defects, which can effectively avoid missed detections, reduce false detections through data preprocessing and data fusion of various model results, and improve image quality through denoising and image enhancement technology; the dynamic model optimization algorithm combined with the micro-buffer is used to improve the stability of the model, so that the chip defect detection system can show a high defect detection accuracy even when facing unknown defects.

[0033] like Figure 2 As shown, the intelligently initialized defect rapid screening and location algorithm is designed with a defect location model. The defect location model includes two sub-networks: a progressive reconstruction model and a detection model. Specifically, the following steps are included:

[0034] Step 1: Use 3D X-ray microscope technology to scan defect-free chip samples (normal chips) at multiple angles to obtain a large number of defect-free two-dimensional images. Then pass All defect-free image data are normalized to [0,1]. nor represents the original 2D image of a defect-free chip, represents an m×n dimensional real matrix; μ represents the mean of the defect-free image data, and σ represents the variance of the defect-free image data.

[0035] Step 2: For the normalized defect-free images, a random number judgment method is used to decide whether to generate pseudo-defect images. The adaptive sampling mechanism is used to control the ratio of defect-free images and pseudo-defect images in the training data. The pseudo-defect images are used as input data for training the progressive reconstruction model. This can reduce the probability of false detection and improve the subsequent defect location accuracy. The progressive reconstruction model trained with pseudo-defect images, i.e., simulated abnormal images, can reconstruct high-quality two-dimensional images, which can improve the speed of defect detection by the detection model.

[0036] The pseudo defect image I def =I nor ·(1-M mask )+N is the random generation of defect mask technology, for the defect-free image I nor Perform operations such as circular occlusion, linear occlusion, and localized corrosion. This method can solve the problems of the low probability of defects in the manufacturing process of semiconductor devices such as chips and the uneven distribution of defects.

[0037] When each image data is input into the defect localization model, a random number r~U(0,1) is generated. When t<T, the probability of a false defect image as input data is If r>Ρ fake , then the defect-free image is used as input data this time, if r≤Ρ fake , then the pseudo defect image is used as input data this time. But when t≥T, the probability of the pseudo defect image being used as input data is P fake (t) = 1.0, the input data are all pseudo defect images. Where r represents a random number between 0 and 1; t represents the current training round number, T represents the total number of training rounds, and P fake Represents the probability value of generating a pseudo-defect image; I def represents the pseudo defect image generated by masking the defect-free area; M mask ∈{0,1} m×n Represents the pseudo defect area mask, 1 represents the defect area, and 0 represents the non-defect area; N~N(0,σ 2 ) represents the Gaussian noise matrix with a mean of 0 and a variance of σ 2 .

[0038] Step 3: The encoder in the progressive reconstruction model starts from the input image I input Extract multi-scale features f k =f enconder,k (I input ), the second and seventh layer output feature maps are used as part of the detection model input. The decoder in the progressive reconstruction model processes the obtained multi-scale features through operations such as transposed convolution and upsampling, and finally obtains the reconstructed image I r =f deconder (f k ). Where, f={f1,f2,…,f K} represents the multi-scale feature representation formed by extracting features layer by layer; f enconder,k (●) indicates the encoder; f deconder (●) represents the decoder, so that the reconstructed image I r As close to a defect-free image as possible nor .

[0039] Step 4: The progressive reconstruction model uses a pixel-level error loss function to calculate the reconstructed image I r and the true defect-free image I nor The pixel difference between the two, the pixel level error loss function expression is:

[0040]

[0041] Use the structural similarity loss function to evaluate the reconstructed image I r and the true defect-free image I nor The structural similarity between them ensures that the texture information of the reconstructed image is not lost. The structural similarity loss function expression is:

[0042]

[0043] Gradient loss is used to ensure that the edge information of the reconstructed image is complete. The gradient loss expression is:

[0044]

[0045] The final combined loss function is L rec =αL MSE +βL SSIM +γL grad , where α, β, and γ are hyperparameters used to control the weights of the above three loss functions.

[0046] Step 5: Input image I input , reconstructed image I r and multi-scale feature maps from the intermediate layers of the progressive reconstruction model network Input into the detection model, in order to enhance the contrast of the defect area, through R diff =|I input -I r |Get the residual image R diff , and the residual image is also input into the detection model. Due to the multi-scale feature map f k With different resolutions h k ×w k , need to pass The resolution is adjusted to the same size m×n. Finally, all processed data are concatenated into a multi-channel tensor Input into the detection model. Here, 3+K represents the number of original image channels (3) plus the number of multi-scale feature layers (K). This feature fusion approach can alleviate the overfitting problem caused by overly obvious defects in artificially generated abnormal images.

[0047] Step 6: The encoder in the detection model uses a convolutional neural network f k =f encoder,k (I final ) to extract defect features of different scales and improve detection accuracy. Use the attention mechanism to weightedly combine features of different scales to improve the detection ability of defects of different sizes and obtain the final comprehensive features. Among them, f k represents the multi-scale feature map of the kth layer; w krepresents the attention weight of the kth layer; W represents the linear transformation matrix used for feature weighting.

[0048] Step 7: The decoder in the detection model uses Unified Perception and Analysis Network Decode and get the defect heat map S score =f discriminative (f fused ), and then binarize the defect heat map to obtain the final defect location result

[0049] Locate the defect area through the defect location results. pred (i, j) is the binary defect mask, 1 represents the defect area, 0 represents the normal area, and T is the defect discrimination threshold.

[0050] The detection model uses the focal loss function to compare the difference between the generated defect heat map and the pseudo defects generated on the defect-free image, and adjusts various parameters in the detection model through the loss function value to enhance the detection capability of small areas.

[0051] Step 8: After the defect location model training is completed, the model is deployed. Figure 3 As shown, multi-angle projections obtained by real-time scanning using 3D X-ray microscope technology As the input of the progressive reconstruction model, no pseudo defect image data is generated. Set the reconstruction threshold T θ , the defect heat map S output by the detection model score Use the Softmax function to normalize and get the reconstruction probability score S per =Softmax(S score ), and then the reconstruction probability score is processed by the maximum pooling to obtain the abnormal score, that is, the defect score θ = Global max pool (Avg.pool (S per )). If θ>T θ , it means that the quality of the reconstructed image is too poor and needs to be reconstructed again. r Input into the progressive reconstruction model and repeat the above steps 2 to 7 until θ≤T θ It represents the reconstructed image I r With the defect-free image I nor Similarly, the image reconstruction is completed. Represents a three-dimensional data tensor, m represents the number of rows of the image, n represents the number of columns of the image, N p Indicates the number of projection images, that is, different angles during X-ray scanning.

[0052] Step 9: End.

[0053] like Figure 4 and Figure 5As shown, the three-dimensional microstructure reconstruction algorithm for multi-angle projection is designed with a high-quality reconstruction model, which can also be called a high-precision reconstruction model, and specifically includes the following steps:

[0054] Step 1: Scan the projection image obtained by 3D X-ray microscope technology Perform normalization operation to obtain normalized projection data Generate high-quality 3D reference volume data using filtered back projection where μ p and σ p is the mean and standard deviation of the projected data; P norm (θ, t) represents the normalized projection data, θ represents the projection angle, and t represents the horizontal coordinate of the projection image; g filter (P norm (θ, t)) represents the result of filtering the projection data; ω(θ, t) represents the weighting coefficient, which is used to balance the contribution of projection data at different angles to image reconstruction.

[0055] Step 2: In order to align the coordinates of the two, first calculate the center of mass of the normalized projection data The centroid of high-quality 3D reference volume data Then perform rigid transformation to obtain projection data and volume data And the data pair after the two are registered (P aligned ,V aligned ). Where P norm (i, j, k) represents the pixel value of the projection data; V true (i, j, k) represents the pixel value of high-quality 3D reference data; d represents the number of layers in the depth direction of the volume data; P aligned and V aligned Represent the registered projection data and volume data respectively; Represents the rotation matrix, which is used to calculate the rotation relationship between the projection coordinate system and the filtered back-projection volume data coordinate system; Where t represents the translation vector, which is used to calculate how much the center of the generated volume data needs to be translated in order to project data on it. This step is used to ensure that P aligned and V aligned Consistency in space.

[0056] Step 3: The registered data pair (P aligned ,V aligned ), defect heat map S score And the defect location result M pred Input into the high-quality reconstruction model, the high-quality reconstruction model passes through the encoder

[0057] f k =f enconder,k(P aligned ), k∈[1, K] extracts multi-scale features, uses multi-layer convolution to operate on feature information such as edges and textures in two-dimensional images, and then gradually upsamples and uses attention mechanisms through the decoder to carefully restore the three-dimensional volume data V r =f decoder (f1,f2,…,f K ). Among them, the high-quality reconstruction model is lost through the mean square error and structural similarity loss To calculate the difference between the reconstructed volume data and the reference volume data, the overall loss L is obtained total =α·L MSE +β·L SSIM , and then iteratively optimize the network weights through gradient descent Where α and β are weight parameters, and θ is a model parameter. This step uses the registered data to train the network model, enabling the network to learn how to generate high-quality 3D volume data from a small amount of 2D radiographic projection data.

[0058] Step 4: Slice the high-quality 3D reference volume data into 2D slices according to depth to obtain high-precision reference data slices F k =Slice(V true ,k),k∈[1,d]. The defect location result M pred Slice F with high-precision reference data k Combine to get the real structure slice Then use structural similarity calculation For the defect heat map area S score and real structure slices Make a comparison.

[0059] Set the false detection threshold T S , if the value of the structural similarity SSIM is lower than the false detection threshold T S , indicating that the defect localization model has produced a false detection. Otherwise, the area marked by the defect heat map is correct. Represents the structural similarity measure, with a value between 0 and 1; μ S ,μ F Defect heat map S score and real structure slices The mean of S ,σ F Defect heat map S score and real structure slices Variance of σ SF represents the covariance between the two; C1 and C2 are small constants to avoid division by zero. This step can not only reduce the probability of false detection through this comparison method, but also improve the accuracy of model reconstruction.

[0060] Step 5: If a false detection occurs, remove the false detection area from the defect location result to obtain the corrected location result. Then remap the corrected positioning results to the defect heat map to generate the corrected defect heat map This improves detection accuracy and reduces the possibility of false detection. If the structural similarity measure is less than the set false detection threshold, it is considered a false detection and the pixel position value in the positioning result is changed to 0; otherwise, the original value remains unchanged.

[0061] Step 6: After the high-precision reconstruction model training is completed, the model is deployed and will no longer use the data pair for training, but will use the actual scanned data P input And the detection result M of the detection model pred (i, j) to select the region of interest for projection data acquisition.

[0062] Step 7: Normalize the scanned data to obtain normalized data The detection results of the detection model and the normalized data are clipped and projected to obtain the clipped projection data P crop =P input ·M pred , the trimmed data P crop and normalized data The data are respectively input into the trained high-quality reconstruction model for reconstruction, and the high-precision reconstruction data V of the local defects are obtained. defect =f recon (P crop ) and the complete reconstruction data

[0063] Step 8: High-quality reconstruction model using high-resolution reconstruction data V of the defect area defect Finally, the reconstructed high-quality 3D volume data is sliced by depth to generate a high-quality 2D image. If no false detection occurs, S final =S score , otherwise

[0064] Step 9: End.

[0065] like Figure 6 As shown, the dynamic model optimization algorithm combined with micro-buffer is designed with a dynamic optimization model, which specifically includes the following steps:

[0066] Step 1: In the initial stage, the defect heat map S final High-precision 2D slices generated from high-quality reconstruction models Store to buffer The buffer is used to efficiently store newly detected abnormal data and defect samples, which can effectively alleviate catastrophic forgetting and improve detection accuracy.

[0067] Then, the defect area is obtained by combining the defect heat map with high-precision two-dimensional slices. Construct the initial sample set and extract the initial sample first by defect area Store the p×p-sized blocks cropped from the initial sample into buffer B init ={F1,F2,…,F s}, then calculate the sample feature f i =f enconder (F i ), select representative samples F by sample characteristics rep =Cluster(f i ),|F rep |=k, and finally combine the initial buffer with the representative sample to ensure that the most representative sample B is stored in the buffer samples =B init ∪F rep . Among them B samples represents the reference buffer, s represents the capacity of the buffer, and m×n represents the two-dimensional resolution of a single sample; Represents a two-dimensional slice containing only the defect area, which helps to reduce the computational complexity of the model and improve classification accuracy; i Indicates the extracted defect sample; B init represents the initial reference buffer, which stores defect samples of known categories; F rep Represents the combination of representative samples, including the k most representative samples.

[0068] Step 2: Dynamically optimize the model to extract features from high-precision two-dimensional slices and obtain high-dimensional feature vectors of defect samples Feature enhancement is performed by combining the high-dimensional feature vector of the defect with the defect heat map. aug =f k ·S final .

[0069] Step 3: The dynamic optimization model uses the enhanced high-dimensional feature vector to calculate the path integral, extract features, and classify defects.

[0070] During the dynamic optimization model training process, the path integral method is used to record the trajectory of each parameter change The model calculates the importance of each parameter by the parameter change trajectory

[0071] The total loss function of the model is in represents the parameter θ k The changing trajectory in the nth round of training, g k (θ(t)) represents the gradient of the parameters, θ' k (t) represents the update direction of the parameters; represents the parameter θ k Of all past missions, represents the parameter change, ξ represents the damping factor to prevent the denominator from being 0.

[0072] The model uses cross entropy loss To learn new categories of defects, D n Represents a dataset of new categories, and f(x) represents the model's predicted probability for sample x.

[0073] The model uses regularization loss To protect old category defects and prevent catastrophic forgetting.

[0074] In this step, the dynamic optimization model batch processes new data stored in the buffer and uses path integrals to calculate the importance of various parameters to continuously optimize the model's defect detection capabilities. When the model encounters a defect type it has never recognized before, it can adjust the weights by comparing the new defect type with existing defect types, allowing the dynamic optimization model to identify the new defect type. At the same time, by rationally combining the newly added data with the original data, the model's ability to recognize old defect types is not lost, effectively mitigating catastrophic forgetting.

[0075] Step 4: If a new category of defects appears, cluster the sample features of the new category of defects and select representative samples F rep =Cluster(f Cn ), add the new category sample to the buffer, and samples =B samples ∪F rep If the buffer overflows, the oldest category sample is deleted to maintain the optimal sample set.

[0076] Step 5: After the dynamic optimization model training is completed, the model is deployed. The input of the model is still the defect heat map and high-precision two-dimensional slices. The two-dimensional slices are feature extracted, and then the defect classification operation is performed to calculate the probability of the defect category. To determine which category the defect belongs to. i |F k ) represents sample C i Belongs to category F k The probability of f classify (F k ,C i ) represents the classification network for Ci The output score; c represents the number of defect categories that the current model has learned.

[0077] Step 6: The dynamic optimization model will select the category with the highest probability based on the calculated probability Output as classification result. Set the category confidence threshold T C , if maxΡ(C i |F k )<T C , then we think that sample F k If a defect is likely a new category, the new category defect features, defect heatmap, and model weights are stored in a buffer to facilitate subsequent model updates. If a new category is detected, the dynamic optimization model is optimized by calculating the model's loss function and adjusting parameter importance to gradually adapt the model to the new category and improve long-term classification performance.

[0078] Step 7: The dynamic optimization model generates a comprehensive defect diagnosis report. This report allows inspectors to analyze defect causes and implement process improvements. The report not only analyzes the defect type, location, and severity, but also assesses the performance impact of the defect on semiconductor device samples, such as chips. The report also displays defects in samples in 3D visualization, helping inspectors quickly identify problems and minimize losses.

[0079] Step 8: End.

[0080] In summary, previous model training relied on a balanced dataset of positive and negative samples. However, defects are typically infrequent in actual industrial manufacturing processes, making it difficult to obtain sufficiently large and diverse datasets. The intelligently initialized fast defect screening and localization algorithm of the present invention uses data augmentation techniques to expand the sample size of normal and simulated abnormal samples. Using an adaptive sampling mechanism and a random number judgment method, it randomly selects non-abnormal images (i.e., defect-free images) or simulated abnormal images (i.e., pseudo-defect images) as input data for the progressive reconstruction model, reducing the probability of false detections and improving the accuracy of subsequent defect localization. After training on simulated abnormal images, the progressive reconstruction model can reconstruct high-quality two-dimensional images, increasing the speed of defect detection by the detection model. Due to the lack of a one-to-one correspondence between the reconstructed image and its original image at the pixel level, abnormality identification methods based on pixel-by-pixel comparison often result in higher false detection rates. To address this issue, a deep learning-based detection model is used to detect abnormal regions at both the feature and semantic levels, thereby improving the accuracy of abnormality detection. Secondly, compared to real defects, the defect regions in simulated abnormal images typically have clear edges and patterns that differ significantly from normal regions. This makes the simulated abnormalities stand out clearly in the image, leading to potential overfitting during the training of the detection model. To address this issue, the detection model not only inputs the original image, the reconstructed image, and the residual image, but also integrates the intermediate-layer feature information obtained during the reconstruction process. This fusion of image and intermediate-layer features serves as input to the detection model, reducing the model's tendency to rely solely on clear boundaries to locate abnormal areas, effectively alleviating the overfitting problem.

[0081] To address the problems of traditional defect detection algorithms, such as their reliance on slice selection and their lack of stereoscopic detection and analysis capabilities, the present invention utilizes a novel 3D microstructure reconstruction algorithm for multi-angle projection. First, preprocessing operations such as denoising and artifact reduction are performed on the 2D projection data obtained from a full scan using 3D X-ray microscopy technology, significantly improving the reconstruction quality of the high-precision reconstruction model. The preprocessed data is then input into the high-precision reconstruction model to generate high-quality 3D volume data. During the training phase, the high-precision reconstruction model spatially registers the high-quality 3D volume data obtained using filtered back projection with the 2D projection data. After training using the registered pairs, the high-precision reconstruction model significantly improves its reconstruction capabilities. By analyzing high-precision local defect reconstructed images and global reconstructed images, the defect detection system can easily detect anomalies such as bubbles, cracks, and solder voids. Secondly, the high-precision reconstruction model compares the defect areas generated by the defect localization model with the reconstructed high-precision areas, reducing the probability of false detections and improving model reconstruction accuracy. Finally, the high-precision reconstruction model stores the complete 3D data, facilitating manual verification by inspectors and providing a rich foundation for subsequent automated testing.

[0082] Most current chip defect detection methods rely on training with fixed data sets, which results in a limited number of defect types that can be learned and poor generalization capabilities of models actually deployed, leading to problems in achieving optimal performance. The dynamic optimization model of the present invention is combined with a buffer to store defect detection-related content such as abnormal area data and two-dimensional data slices in the buffer, and adjusts model parameters by batch processing the data in the buffer to improve the generalization capabilities of the model. Secondly, traditional models often suffer from catastrophic forgetting problems during training, but the algorithm that uses a dynamic optimization model combined with a buffer can effectively alleviate catastrophic forgetting and improve detection accuracy, because the buffer can efficiently store newly detected abnormal data and defect samples, and give priority to preserving representative abnormal data and defect samples. The trained dynamic optimization model greatly improves the ability to recognize new defects, and even when dealing with unprecedented defect types, the dynamic optimization model can still perform well.

[0083] In the field of defect detection of semiconductor devices such as chips, real-time detection and processing capabilities are related to the production capacity of semiconductor devices. Only by timely discovering the defects of the currently scanned device can the corresponding pipeline adjustments be made to prevent the factory from suffering large losses. The present invention quickly locates the possible locations of defects through an intelligently initialized defect rapid screening and positioning algorithm. Then, a three-dimensional microstructure reconstruction algorithm for multi-angle projection performs a cropping operation on the possible locations of defects, and uses the cropped slices for high-precision reconstruction. The results of the high-precision reconstruction are compared with the cropped slices to determine whether a false detection has occurred, thereby providing rapid feedback for the chip production line to produce chips. Finally, the dynamic model optimization algorithm combined with the micro-buffer uses a storage area to store the most representative defect samples. Processing the most representative samples each time can not only alleviate catastrophic forgetting, but also improve the real-time performance and detection capability of the chip defect detection system, while also greatly reducing the system's data processing time.

[0084] Also, in the field of defect detection of semiconductor devices such as chips, the accuracy is directly related to many aspects such as the quality control of the production line, the reliability of semiconductor devices, and cost control. However, in actual applications, there are many factors that affect the accuracy of defect detection, mainly including missed detection, false detection, and noise interference. The present invention reduces the location of defects through an intelligently initialized defect rapid screening and positioning algorithm, thereby improving the speed and ability of defect detection; it captures subtle defects through a three-dimensional microstructure reconstruction algorithm for multi-angle projection, which can effectively avoid missed detection, reduce false detection through data preprocessing and data fusion of various model results, and improve image quality through denoising and image enhancement technology; and improves the stability of the model by using a dynamic model optimization algorithm combined with a micro-buffer, so that the chip defect detection system can show a high defect detection accuracy even when facing unknown defects. By comprehensively applying the above-mentioned technical means, the chip defect detection system can still stably and reliably perform chip defect detection tasks and maintain a high accuracy rate even in a complex and changeable production environment.

[0085] The above description is only a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the implementation process of the present invention is described in detail above, it is still possible for those familiar with the art to modify the technical solutions described in the above examples or to replace some of the technical features therein with equivalents. Any modifications, equivalent replacements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A chip defect detection method based on multi-stage feature fusion and dynamic optimization, which is designed with a defect localization model, a high-quality reconstruction model and a dynamic optimization model, characterized in that: include: The defect localization model is used to perform data preprocessing, image reconstruction, and feature extraction and fusion operations on the defect-free image of the chip to generate a defect heat map and binarize it to obtain the defect localization result; The normalized projection data, defect heat map and defect location results of the chip are input into a high-quality reconstruction model. The high-quality reconstruction model crops and projects the defect location results and normalized projection data, and then performs reconstruction and recovery processing to obtain local defect reconstruction data and high-quality three-dimensional volume data. The high-quality three-dimensional volume data is then deep sliced to obtain high-precision two-dimensional slices. The defect heat map and the local defect reconstruction data are compared to correct the defect heat map. Using a dynamic optimization model, defect heat maps and high-precision two-dimensional slices are combined into defect samples and stored in a buffer. New categories of defects that appear are learned and identified to update the defect samples in the buffer.

2. The chip defect detection method according to claim 1, wherein: Use 3D X-ray microscope technology to scan defect-free chip samples (normal chips) at multiple angles to obtain defect-free images. Then pass All defect-free image data are normalized to [0,1], where I nor represents the original defect-free image, represents an m×n dimensional real matrix; μ represents the mean of the defect-free image data, and σ represents the variance of the defect-free image data.

3. The chip defect detection method according to claim 2, wherein: The defect location model includes a progressive reconstruction model. For the normalized defect-free image, a random number judgment method is used to determine whether to generate a pseudo-defect image. An adaptive sampling mechanism is used to control the ratio of the defect-free image and the pseudo-defect image in the input data, and the pseudo-defect image is used as the input image data of the progressive reconstruction model. input The pseudo defect image is a random defect mask generated by the defect-free image I nor Perform circular occlusion, linear occlusion and local erosion operations to generate.

4. The chip defect detection method according to claim 3, wherein: The encoder in the progressive reconstruction model starts from the input image data I input Extract multi-scale features f k =f enconder,k (I input ), the decoder in the progressive reconstruction model processes the obtained multi-scale features through operations such as transposed convolution and upsampling, and finally obtains the reconstructed image I r =f deconder (f k ), f={f1,f2,…,f K } represents the multi-scale feature representation formed by extracting features layer by layer; f enconder,k (●) indicates the encoder; f deconder (●) represents the decoder, so that the reconstructed image I r As close to a defect-free image as possible nor ; Among them, the progressive reconstruction model uses a pixel-level error loss function to calculate the reconstructed image I r and the true defect-free image I nor The pixel difference between them is used to evaluate the reconstructed image I using the structural similarity loss function. r and the true defect-free image I nor The structural similarity between the two images is calculated, and the gradient loss is used to ensure the edge information of the reconstructed image is complete.

5. The chip defect detection method according to claim 4, wherein: The defect location model also includes a detection model, which inputs the image I input , reconstructed image I r , residual image R diff and multi-scale feature maps from the intermediate layers of the progressive reconstruction model network After being adjusted to the same size, the residual image R diff By R diff =|I input -I r |Get, the encoder in the detection model uses a convolutional neural network f k =f encoder,k (I final ) to extract defect features of different scales, and use the attention mechanism to weightedly combine features of different scales to obtain the final comprehensive features Among them, f k represents the multi-scale feature map of the kth layer; w k Represents the attention weight of the kth layer; W represents the linear transformation matrix, which is used for feature weighting; the decoder in the detection model uses a unified perceptual parsing network for decoding to obtain the defect heat map S score =f discriminative (f fused ), and then binarize the defect heat map to obtain the final defect location result Locate the defect area through the defect location results, where M pred (i, j) is the binary defect mask, 1 represents the defect area, 0 represents the normal area, and T is the defect discrimination threshold.

6. The chip defect detection method according to claim 5, wherein: Set the reconstruction threshold T θ , the defect heat map S output by the detection model score Use the Softmax function to normalize and get the reconstruction probability score S per =Softmax(S score ), and then the reconstruction probability score is processed by the maximum pooling to obtain the abnormal score, that is, the defect score θ = Globalmaxpool(Avg.pool(S per )); if θ>T θ , then reconstruct again, and reconstruct the image I r Input into the progressive reconstruction model until θ≤T θ It represents the reconstructed image I r With the defect-free image I nor Similarly, image reconstruction is completed, where Represents a three-dimensional data tensor, m represents the number of rows of the image, n represents the number of columns of the image, N p Indicates the number of projection images, that is, different angles during X-ray scanning.

7. The chip defect detection method according to claim 1, wherein: Normalize the projection image obtained after a comprehensive scan of the chip sample using 3D X-ray microscope technology to obtain normalized projection data Then use filtered back projection to generate high-quality 3D reference volume data where μ p and σ p is the mean and standard deviation of the projected data; P norm (θ, t) represents the normalized projection data, θ represents the projection angle, and t represents the horizontal coordinate of the projection image; g filter (P norm (θ, t)) represents the result of filtering the projection data; ω(θ, t) represents the weighting coefficient, which is used to balance the contribution of projection data at different angles to image reconstruction; The centroid of the normalized projection data and the centroid of the high-quality three-dimensional reference volume data are calculated respectively, and then a rigid transformation is performed to obtain the projection data and volume data, as well as the data pair after the two are registered.

8. The chip defect detection method according to claim 7, wherein: The high-quality reconstruction model inputs the registered data pair, the defect heat map and the defect localization result, extracts multi-scale features through the encoder, utilizes multi-layer convolution to operate the feature information about edges and textures in the two-dimensional image, and then gradually upsamples and uses the attention mechanism of the decoder to restore the three-dimensional volume data to obtain local defect reconstruction data and high-quality three-dimensional volume data; then, a structural similarity calculation method is used to compare the defect heat map area and the local defect reconstruction data, and a false detection threshold is set. If the value of the structural similarity is lower than the false detection threshold, it means that the defect localization model has generated a false detection. The falsely detected area is then removed from the defect localization result to obtain a corrected localization result, and the corrected localization result is remapped to the defect heat map to generate a corrected defect heat map.

9. The chip defect detection method according to claim 1, wherein: The dynamic optimization model stores the corrected defect heat map and high-precision two-dimensional slices in a buffer, and combines the defect heat map with the high-precision two-dimensional slices to obtain the defect area, and constructs an initial sample set based on the defect area; Then, the initial sample is extracted and cut into small blocks and stored in the buffer. The sample features of each block are calculated, and representative samples are selected based on the sample features. Finally, the initial buffer is combined with the representative samples so that the most representative samples are stored in the buffer. The dynamic optimization model then extracts features from high-precision two-dimensional slices to obtain high-dimensional feature vectors of the defect samples. This feature vector is then combined with the defect heat map to enhance the features. The enhanced high-dimensional feature vector is then used to calculate path integrals, extract features, and classify defects. If a new category of defects appears, the sample features of the new category of defects are clustered, representative samples are selected, the new category samples are added to the buffer, and the buffer is updated. If the buffer overflows, the oldest category samples are deleted to maintain the optimal sample set.

10. The chip defect detection method according to claim 9, wherein: During the dynamic optimization model training process, the path integral method is used to record the trajectory of each parameter change. The importance of each parameter is calculated through the parameter change trajectory. The cross entropy loss is used to learn new category defects, and the regularization loss is used to protect old category defects.

Images