An advanced process interface contamination automatic identification method and device

Abstract

The present application relates to the technical field of semiconductor advanced process detection and failure analysis, and particularly relates to an advanced process interface contamination automatic identification method and device, comprising: obtaining scanning electron microscope images, transmission electron microscope images and energy spectrum data of a target interface region; performing a spatial alignment operation on the multi-modal data to obtain multi-modal aligned data in a unified coordinate system; extracting texture features, contrast features and element features from the aligned data; inputting the above features into a pre-trained classification model to identify suspected contamination regions and determine contamination categories and confidence levels; outputting contamination categories, position information and confidence levels, and writing the identification results back to a case library for subsequent model training or review calling. The present application realizes automatic fusion and joint analysis of multi-modal data, and improves the efficiency, accuracy and consistency of interface contamination identification.

Description

Technical Field

[0001] This invention relates to the field of semiconductor advanced process inspection and failure analysis technology, and in particular to an automatic identification method and apparatus for interface contamination in advanced processes. Background Technology

[0002] As the process nodes of advanced logic, advanced memory, and advanced packaging products continue to evolve, the internal structure of chips is becoming increasingly complex. Metal interconnect interfaces, bonding interfaces, and buried layer interfaces have become high-risk areas for contaminants. Interface contamination typically manifests as nanoscale hydrocarbon residues, metal particles, or oxide spots, exhibiting different contrasts, textures, and elemental characteristics in scanning electron microscopy (SEM) images, transmission electron microscopy (TEM) images, and energy dispersive spectroscopy (EDS) data. In failure analysis practice, analysts need to simultaneously observe the overall morphology of SEM images, the high-resolution structure of TEM images, and the elemental distribution of EDS data to comprehensively determine the presence, type, and extent of contamination.

[0003] However, existing technologies lack a unified spatial alignment and automatic correlation identification method for SEM images, TEM images, and energy dispersive spectroscopy (EDS) data. Analysts typically rely on manual comparison of the three types of data, manually marking suspicious locations between images, and judging the contamination category based on experience. This approach is not only inefficient, but also suffers from poor consistency and reproducibility in contamination identification results due to differences in judgment standards among different operators. This is especially true when contaminants are small, have weak contrast, or are similar to the background structure, making manual interpretation prone to missed detections and misjudgments. Summary of the Invention

[0004] To overcome the above deficiencies, this invention provides an automatic identification method and apparatus for interface contamination in advanced processes. It aims to improve the problems in the prior art, which lacks automatic spatial alignment and joint analysis of scanning electron microscope images, transmission electron microscope images and energy dispersive spectroscopy data, resulting in low efficiency, poor consistency and inability to form traceable conclusions in interface contamination identification.

[0005] In a first aspect, the present invention provides the following technical solution: an automatic identification method for interface contamination in advanced manufacturing processes, comprising the following steps:

[0006] Acquire scanning electron microscope images, transmission electron microscope images, and energy dispersive spectroscopy data of the target interface area;

[0007] Spatial alignment operations are performed on the scanning electron microscope image, the transmission electron microscope image, and the energy dispersive spectroscopy data to obtain multimodal aligned data in a unified coordinate system.

[0008] Extract texture features, contrast features, and element features from the multimodal aligned data;

[0009] The texture features, contrast features, and element features are input into a pre-trained classification model to identify suspected contamination areas in the target interface region and determine the contamination category and confidence level of the suspected contamination areas.

[0010] Output the pollution category, the location information of the suspected polluted area, and the confidence level, and write the identification results back to the case library for subsequent model training or verification.

[0011] Preferably, the step of acquiring scanning electron microscope images, transmission electron microscope images, and energy dispersive spectroscopy data of the target interface region specifically includes:

[0012] At least three shared marker points are prepared in the target interface region, and the shared marker points can be imaged and identified under both scanning electron microscopy and transmission electron microscopy.

[0013] Scanning electron microscope images, transmission electron microscope images, and energy spectrum surface scan data or energy spectrum spot scan data covering the target interface region are acquired respectively.

[0014] Record the spatial coordinate information of the scanning electron microscope image, the transmission electron microscope image, and the energy dispersive spectroscopy data in the image coordinate system.

[0015] Preferably, the step of obtaining multimodal aligned data in a unified coordinate system specifically includes:

[0016] Extract the pixel coordinates of at least three shared marker points from the scanning electron microscope image and the transmission electron microscope image, respectively;

[0017] Calculate the spatial transformation matrix based on the pixel coordinate correspondence of the shared marker points in the scanning electron microscope image and the transmission electron microscope image;

[0018] The transmission electron microscope image is transformed to the coordinate system of the scanning electron microscope image using the spatial transformation matrix to obtain an aligned transmission electron microscope image;

[0019] The acquisition coordinates of the energy spectrum data are mapped to the coordinate system of the scanning electron microscope image through a calibration relationship to obtain aligned energy spectrum data;

[0020] The scanning electron microscope image, the aligned transmission electron microscope image, and the aligned energy dispersive spectroscopy data are collectively used as the multimodal alignment data.

[0021] Preferably, the steps of extracting texture features, contrast features, and element features specifically include:

[0022] Within the image space of the multimodal alignment data, a sliding window is used to traverse the target interface region;

[0023] For each sliding window, texture and contrast features are extracted from the aligned transmission electron microscope image or the scanning electron microscope image, and elemental features are extracted from the aligned energy spectroscopy data.

[0024] The texture features, contrast features, and element features extracted from each sliding window are associated and stored according to the window position to form a feature vector set corresponding to the window position.

[0025] Preferably, the step of identifying suspected contaminated areas in the target interface region specifically includes:

[0026] The feature vectors corresponding to each sliding window are input into the pre-trained classification model, and the classification model outputs the pollution category prediction result for each sliding window.

[0027] Based on the pollution category prediction results, adjacent sliding windows with the same pollution category are merged into connected components to obtain at least one connected region.

[0028] The connected region is identified as the suspected contaminated region.

[0029] Preferably, the step of determining the pollution category and confidence level of the suspected contaminated area specifically includes:

[0030] Obtain the pollution category prediction results and confidence values ​​for each sliding window constituting the suspected contaminated area;

[0031] The frequency of each category in all sliding windows is counted, and the pollution category with the highest frequency is taken as the pollution category of the suspected polluted area.

[0032] A statistic is calculated based on the confidence values ​​of all the sliding windows, and the statistic is used as the confidence level of the suspected contaminated area.

[0033] Preferably, the step of writing the recognition results back to the case library for subsequent model training or verification specifically includes:

[0034] The boundary coordinates or contour vertex coordinates of the suspected contaminated area in the coordinate system of the scanning electron microscope image are used as the location information;

[0035] The pollution category, the location information, and the confidence level are associated and stored as a structured identification record;

[0036] The structured recognition record is output to a display terminal, storage medium, or downstream analysis system;

[0037] The structured recognition records, along with the corresponding scanning electron microscope images, transmission electron microscope images, and energy dispersive spectroscopy data, are written into a case library for updating the training set of the classification model or for use in manual review tasks.

[0038] Secondly, the present invention provides the following technical solution: an automatic identification device for interface contamination in advanced manufacturing processes, the device comprising:

[0039] The data acquisition module is used to acquire scanning electron microscope images, transmission electron microscope images, and energy dispersive spectroscopy data of the target interface area, and send the acquired data to the spatial alignment module.

[0040] The spatial alignment module, connected to the data acquisition module, is used to receive the scanning electron microscope image, the transmission electron microscope image, and the energy dispersive spectroscopy data, and perform spatial alignment operations on them to obtain multimodal aligned data in a unified coordinate system, and then send the multimodal aligned data to the feature extraction module.

[0041] The feature extraction module, connected to the spatial alignment module, is used to receive the multimodal alignment data, extract texture features, contrast features, and element features from the multimodal alignment data, and then send the texture features, contrast features, and element features to the contamination identification module.

[0042] The contamination identification module, connected to the feature extraction module, is used to receive the texture features, the contrast features, and the element features, and input them into a pre-trained classification model to identify suspected contamination areas in the target interface area, determine the contamination category and confidence level of the suspected contamination area, and then send the contamination category, the location information of the suspected contamination area, and the confidence level to the output write-back module.

[0043] The output write-back module, connected to the pollution identification module, is used to receive the pollution category, the location information of the suspected pollution area, and the confidence level, output the pollution category, the location information of the suspected pollution area, and the confidence level, and write the identification results back to the case library for subsequent model training or verification.

[0044] The present invention has the following beneficial effects:

[0045] 1. In this invention, by establishing a unified coordinate system, scanning electron microscope images, transmission electron microscope images, and energy dispersive spectroscopy data are automatically spatially aligned. Based on the aligned multimodal data, texture, contrast, and elemental features are automatically extracted for classification and identification, avoiding manual comparison and subjective interpretation. This significantly improves the automation level and consistency of the interface contamination identification process and solves the problems of low analysis efficiency and poor consistency caused by the lack of a unified multimodal association mechanism in the prior art.

[0046] 2. In this invention, a combination of multimodal feature fusion and classification model is adopted. Compared with conventional methods of single-modal or manual splicing, it can more comprehensively characterize the image structure and element composition information of the polluted area, effectively improve the accuracy of pollution identification, and reduce the risk of missing weak pollution signals, so that the identification precision and recall rate are significantly improved.

[0047] 3. In this invention, a pre-trained classification model is used to comprehensively judge the fused features, and the final pollution category is determined by combining the regional voting mechanism after merging connected components. This reduces the misclassification of categories caused by insufficient information from a single feature, making the output of the pollution category clearer and more reliable, and effectively suppressing the misclassification rate. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the architecture of an automatic interface contamination identification device for advanced manufacturing processes proposed in this invention.

[0049] Figure 2 This is a flowchart illustrating an automatic identification method for interface contamination in advanced manufacturing processes proposed in this invention.

[0050] Figure 3 This is a schematic diagram of the multimodal data acquisition process for an automatic identification method for interface contamination in advanced manufacturing processes proposed in this invention.

[0051] Figure 4 This is a schematic diagram of the multimodal spatial alignment process of an advanced process interface contamination automatic identification method proposed in this invention.

[0052] Figure 5 This is a schematic diagram of the feature extraction process of an automatic identification method for interface contamination in advanced manufacturing processes proposed in this invention.

[0053] Figure 6 This is a schematic diagram of the contamination identification and confidence determination process of an automatic interface contamination identification method for advanced processes proposed in this invention.

[0054] Figure 7 This is a schematic diagram of the result output and write-back process of an advanced process interface contamination automatic identification method proposed in this invention.

[0055] Among them, 110 is the data acquisition module; 120 is the spatial alignment module; 130 is the feature extraction module; 140 is the contamination identification module; 150 is the output write-back module; and 160 is the case library. Detailed Implementation

[0056] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the overall architecture of a device according to an embodiment of the present invention. The present invention provides an implementation of an automatic interface contamination identification device for advanced processes. The device is deployed in a hardware environment of one or more servers, the hardware environment including a data server for data storage and management, a computing server (e.g., a server configured with a graphics processing unit (GPU)) for performing algorithm model training and inference, and an application server for processing user requests and business logic; the user interacts with the device through a client device.

[0058] In one embodiment of the present invention, the device may functionally include multiple modules that work together.

[0059] The 110 data acquisition module is used to acquire scanning electron microscope (SEM) images, transmission electron microscope (TEM) images, and energy dispersive spectroscopy (EDS) data of the target interface region, and then sends the acquired data to the 120 spatial alignment module. In one specific embodiment, the 110 data acquisition module first receives instructions from the user or automated scanning path, controlling the SEM and TEM to acquire high-resolution images of the target interface region, and simultaneously acquires EDS surface scan data or EDS point scan data of the corresponding region using an EDS instrument (such as EDS or EELS). To facilitate subsequent spatial alignment, at least three shared marker points are pre-prepared near the interface during the sample preparation stage (e.g., by depositing platinum or carbon using a focused ion beam). These marker points are clearly visible under both SEM and TEM. While acquiring images and EDS data, the 110 data acquisition module automatically records the spatial coordinate information (e.g., pixel coordinates or stage coordinates) of each data point, and packages the raw data and coordinate information together and sends them to the 120 spatial alignment module.

[0060] A spatial alignment module (120) is connected to a data acquisition module (110) and is used to receive scanning electron microscope (SEM) images, transmission electron microscope (TEM) images, and energy dispersive spectroscopy (EDS) data. It performs spatial alignment operations on these images to obtain multimodal aligned data in a unified coordinate system, and then sends the multimodal aligned data to a feature extraction module (130). In one specific embodiment, the spatial alignment module (120) first extracts the pixel coordinates of at least three shared marker points from the received SEM and TEM images, respectively. Then, based on these corresponding point pairs, it calculates a spatial transformation matrix (e.g., an affine transformation matrix or a projection transformation matrix). Next, it uses this spatial transformation matrix to transform the TEM image to the coordinate system of the SEM image, obtaining an aligned TEM image. Simultaneously, the acquisition coordinates of the EDS data are transformed to the same coordinate system using a pre-calibrated device coordinate mapping relationship, obtaining aligned EDS data. Finally, the aligned TEM image, aligned EDS data, and the original SEM image are combined and output as multimodal aligned data to the feature extraction module (130).

[0061] The 130 feature extraction module is connected to the 120 spatial alignment module. It receives multimodal alignment data and extracts texture features, contrast features, and elemental features from the data. These features are then sent to the 140 pollution identification module. In one specific embodiment, the 130 feature extraction module first uses a sliding window (e.g., a window size of 64×64 pixels with a step size of 32 pixels) to traverse the entire target interface area within the image space of the multimodal alignment data. For each sliding window, texture features and contrast features are extracted from the aligned transmission electron microscope (or scanning electron microscope) image (or image), and elemental features are extracted from the aligned energy spectrum data. Texture features include at least the contrast, energy, and inverse moment statistics of the gray-level co-occurrence matrix; contrast features include at least the standard deviation or skewness of the gray-level values ​​within the window; and elemental features include at least preset feature elements, such as the net peak count or peak-to-background ratio of the energy spectrum for carbon, oxygen, fluorine, and copper. After extraction, the 130 feature extraction module associates and stores the texture features, contrast features, and element features corresponding to each sliding window according to the window position, forming a feature vector set that corresponds one-to-one with the window position, and sends it to the 140 contamination identification module.

[0062] The 140 pollution identification module, connected to the 130 feature extraction module, receives texture features, contrast features, and element features, inputs them into a pre-trained classification model, identifies suspected pollution areas in the target interface region, determines the pollution category and confidence level of the suspected pollution areas, and then sends the pollution category, location information of the suspected pollution areas, and confidence level to the 150 output write-back module. In one specific embodiment, the 140 pollution identification module first inputs each feature vector in the received feature vector set into a pre-trained classification model, such as a random forest, support vector machine, or lightweight neural network. The classification model outputs the probability that each sliding window belongs to each preset pollution category, such as carbon pollution, metal pollution, particulate pollution, and no pollution. The 140 pollution identification module uses the pollution category corresponding to the highest probability value as the preliminary identification result of the sliding window and uses the highest probability value as the confidence level of the window. Then, a connected component merging operation is performed on adjacent sliding windows with the same preliminary identification result to obtain at least one connected region, each connected region being a suspected pollution area. For each suspected contaminated area, the 140 contamination identification module further counts the frequency of contamination categories across all windows within it, and uses the category with the highest frequency as the final contamination category for that area. Simultaneously, it calculates a statistic (e.g., mean, maximum, or median) based on the confidence values ​​of all windows within it, and uses this statistic as the final confidence level for that area. Finally, the contamination category, location information (e.g., area boundary coordinates or contour vertex coordinates), and confidence level of each suspected contaminated area are sent to the 150 output write-back module.

[0063] The 150 output write-back module connects to the 140 pollution identification module. It receives the pollution category, location information of suspected contaminated areas, and confidence levels; outputs the pollution category, location information of suspected contaminated areas, and confidence levels; and writes the identification results back to the 160 case library for subsequent model training or verification. In one specific implementation, the 150 output write-back module first converts the received location information of suspected contaminated areas into boundary coordinates or contour vertex coordinates in the scanning electron microscope image coordinate system. Then, it associates and stores the pollution category, location information, and confidence levels as a structured identification record (e.g., JSON format or database record). The 150 output write-back module outputs this structured identification record to a display terminal (e.g., a graphical user interface), storage medium, or downstream analysis system for user viewing or further processing. Simultaneously, the 150 output write-back module writes the structured identification record, along with the corresponding original scanning electron microscope image, transmission electron microscope image, and energy dispersive spectroscopy data, into the 160 case library.

[0064] The 160-case library connects to the 150-output write-back module to receive and store structured identification records along with corresponding raw scanning electron microscope (SEM) images, transmission electron microscope (TEM) images, and energy dispersive spectroscopy (EDS) data. The identification results accumulated in the 160-case library can be periodically used to update the training set of the classification model in the 140-contamination identification module, or manually reviewed to correct erroneous identifications and re-added to training, thereby achieving incremental learning and performance optimization of the model. When subsequent analysis tasks require review or training, the 140-contamination identification module can retrieve historical data and its annotation results from the 160-case library to improve the identification accuracy of the classification model.

[0065] See attached document Figure 2 , Figure 2 This is a flowchart of an automatic interface contamination identification method for advanced manufacturing processes according to an embodiment of the present invention. The present invention provides an automatic interface contamination identification method for advanced manufacturing processes, which may include the following steps:

[0066] S101. Acquire SEM image, TEM image and energy spectrum data of the target interface region;

[0067] S102. Perform multimodal data spatial alignment to obtain multimodal aligned data in a unified coordinate system;

[0068] S103. Extract texture features, contrast features, and element features from multimodal alignment data;

[0069] S104. Input the features into the pre-trained classification model to identify suspected contaminated areas and determine the contamination category and confidence level;

[0070] S105, output pollution category, location information and confidence level, and write the identification results back to the case library.

[0071] The following will provide a detailed explanation of each step in the AI-based surveying case similarity matching and recommendation method provided in the embodiments of the present invention.

[0072] See attached document Figure 2 The figure is a flowchart of a method according to an embodiment of the present invention. Step S101 of the method will be described in detail below.

[0073] 101-1, such as Figure 3 As shown, at least three shared markers were prepared in the target interface region: At least three shared markers were prepared near the target interface region using focused ion beam deposition. The shared markers were made of platinum or carbon, each marker was circular with a diameter of 50 to 100 nanometers, and the spacing between adjacent markers was 1 to 2 micrometers. The shared markers exhibited identifiable morphologies with different background contrast under both scanning electron microscopy and transmission electron microscopy, which were used for spatial alignment of subsequent multimodal data.

[0074] 101-2. Acquire scanning electron microscope (SEM) images, transmission electron microscope (TEM) images, and energy dispersive spectroscopy (EDS) surface scan data or EDS point scan data covering the target interface region, respectively: Set the accelerating voltage of the SEM to 3 kV to 5 kV and the magnification to 50 kx to 150 kx, and acquire SEM images of the target interface region with an image resolution of 1024 × 1024 pixels. Set the accelerating voltage of the TEM to 200 kV and the magnification to 80 kx to 150 kx, and acquire TEM images of the same target interface region with the same image resolution of 1024 × 1024 pixels. Acquire EDS data covering the target interface region using an EDS spectrometer, either as surface scan data or EDS point scan data. If surface scan is used, the scan step size should match the pixel size of the SEM image. If point scan is used, select at least 9 points covering the shared marker point and surrounding area for acquisition. Set the dwell time for EDS data acquisition to 30 ms to 45 ms to ensure the statistical stability of elemental characteristic peak counting.

[0075] 101-3. Record the spatial coordinates of scanning electron microscope (SEM) images, transmission electron microscope (TEM) images, and energy dispersive spectroscopy (EDS) data in their respective image coordinate systems: For SEM and TEM images, record the coordinates of each pixel in its respective image coordinate system. The image coordinate system has its origin at the top left corner of the image, with the positive X-axis pointing horizontally to the right and the positive Y-axis pointing vertically downwards. The unit of measurement is pixels. For EDS data, record the physical coordinates of each acquisition point or pixel in the stage coordinate system, and record the calibration relationship between the SEM image coordinate system and the stage coordinate system. The calibration relationship is provided by the equipment manufacturer in the calibration document and is expressed as a linear transformation matrix. ,satisfy:

[0076] ;

[0077] in, (u,v) represents the physical coordinates in the stage coordinate system, in micrometers; (u,v) represents the pixel coordinates in the scanning electron microscope image coordinate system. This calibration relationship allows the acquisition location of the energy dispersive spectroscopy (EDS) data to be mapped to the scanning electron microscope image coordinate system for subsequent spatial alignment operations.

[0078] After completing step 101, scanning electron microscope (SEM) images, transmission electron microscope (TEM) images, and energy dispersive spectroscopy (EDS) data of the target interface region are obtained, along with their respective spatial coordinate information. Based on this, step 102 further performs spatial alignment operations on the aforementioned multimodal data to eliminate coordinate differences between different imaging devices, specifically including the following sub-steps:

[0079] 102-1, such as Figure 4 As shown, the pixel coordinates of at least three shared marker points are extracted from the scanning electron microscope (SEM) image and the transmission electron microscope (TEM) image, respectively: The pixel coordinates of the at least three shared marker points prepared in step 101-1 in the SEM image are read and denoted as: ,in, Simultaneously, the pixel coordinates of the same shared marker point in the transmission electron microscope image are read and denoted as... The extraction method is either manual point selection or automatic corner detection algorithm.

[0080] 102-2. Based on the pixel coordinate correspondence of shared marker points in scanning electron microscope (SEM) and transmission electron microscope (TEM) images, calculate the spatial transformation matrix: An affine transformation model is used to describe the spatial mapping relationship between the SEM and TEM images. The affine transformation matrix H satisfies:

[0081] ;

[0082] in, These are the pixel coordinates of a point in a transmission electron microscope image. Given the corresponding pixel coordinates in the scanning electron microscope image coordinate system, use at least three pairs of corresponding point coordinates obtained in step 102-1 to solve for the six unknown parameters of matrix H using the least squares method. .

[0083] 102-3. Transform the transmission electron microscope image to the coordinate system of the scanning electron microscope image using a spatial transformation matrix to obtain the aligned transmission electron microscope image: For each pixel coordinate in the transmission electron microscope image... Based on the affine transformation matrix H obtained in step 102-2, calculate its mapped coordinates in the scanning electron microscope image coordinate system. Since the mapped coordinates may be non-integer, bilinear interpolation is used to calculate the grayscale values ​​of the mapped pixels, resulting in the aligned transmission electron microscope image, denoted as . .

[0084] 102-4. Map the acquisition coordinates of the energy dispersive spectroscopy (EDS) data to the coordinate system of the scanning electron microscope (SEM) image using calibration relationships to obtain aligned EDS data: For each acquisition point or pixel of the EDS data, denote its physical coordinates in the stage coordinate system as... Using the calibration matrix recorded in step 101-3 Map it to the scanning electron microscope image coordinate system:

[0085] ;

[0086] in, These are the aligned coordinates of the energy spectrum data in the scanning electron microscope image coordinate system. The aligned coordinates are then associated with and stored along with the energy spectrum intensity values ​​(or the net count of characteristic peaks for each element), resulting in the aligned energy spectrum data, denoted as . .

[0087] 102-5. Combine the scanning electron microscope image, the aligned transmission electron microscope image, and the aligned energy dispersive spectroscopy data as multimodal aligned data: Use the original scanning electron microscope image... The aligned transmission electron microscope image obtained in step 102-3 and the aligned energy spectrum data obtained in step 102-4 The data are combined into a multimodal aligned dataset under a unified scanning electron microscope image coordinate system. This is used in subsequent feature extraction steps.

[0088] After completing step 102, multimodal aligned data in a unified coordinate system is obtained, including scanning electron microscope images, aligned transmission electron microscope images, and aligned energy dispersive spectroscopy (EDS) data. Based on this, step 103 further extracts texture features, contrast features, and elemental features for contamination identification from the multimodal aligned data, specifically including the following sub-steps:

[0089] 103-1, such as Figure 5 As shown, within the image space of the multimodal aligned data, a sliding window is used to traverse the target interface region: the width of the sliding window is set to W pixels, the height to H pixels, and the horizontal step size is... pixels, vertical step size Pixels. In this embodiment, the pixel is taken. Starting from the top left corner of the scanning electron microscope image, the sub-regions covered by each window are extracted sequentially, first from left to right and then from top to bottom. For each window, the coordinates of its center point in the scanning electron microscope image coordinate system are recorded. , and the row index r and column index c of the window in the image.

[0090] 103-2. For each sliding window, extract texture and contrast features from the aligned transmission electron microscope (TEM) or scanning electron microscope (SEM) image, and extract elemental features from the aligned energy dispersive spectroscopy (EDS) data: For the current sliding window, perform the following three feature extraction methods respectively:

[0091] 103-2-1. Texture Feature Extraction: Extract texture features within a window from the aligned transmission electron microscope image. The gray-level co-occurrence matrix (GLCM) method is used to calculate the GLCM G of the grayscale image within the window. Assume the grayscale levels within the window are quantized to L=16 levels, with pixel pair directions of 0°, 45°, 90°, and 135°, and pixel spacing d=1. For each direction, calculate the contrast C, energy E, and inverse difference moment H:

[0092] ;

[0093] ;

[0094] ;

[0095] The arithmetic mean of the contrast, energy, and inverse difference moment in the four directions is taken to obtain the 3D texture feature vector. .

[0096] 103-2-2 Contrast Feature Extraction: Extract contrast features within a window from aligned scanning electron microscope (SEM) or transmission electron microscope (TEM) images. Calculate the standard deviation of the gray levels of all pixels within the window. and skewness :

[0097] ;

[0098] ;

[0099] in, Let be the total number of pixels within the window, and μ be the mean gray level within the window. The standard deviation and skewness are combined to form a 2D contrast feature vector. .

[0100] 103-2-3. Element Feature Extraction: Extract element features at corresponding positions within the window from the aligned energy spectrum data. Based on the aligned energy spectrum data obtained in step 102-4, obtain the coordinates of the window center point. The representative spectral line is the energy spectrum line at a given location, or the average of the energy spectra of all pixels within the window. The net count of characteristic peaks of preset characteristic elements is extracted from the spectral line. The preset characteristic elements include at least carbon, oxygen, fluorine, and copper, corresponding to peak energy positions of approximately 0.277 keV, 0.525 keV, 0.677 keV, 0.930 keV, and 8.048 keV (copper Kα), respectively. The net count of each element is obtained by subtracting the background continuum spectrum. In addition, the carbon-copper peak-to-background ratio was calculated. The above five values ​​form an elemental feature vector. .

[0101] 103-3. The texture features, contrast features, and element features extracted from each sliding window are associated and stored according to the window position, forming a feature vector set corresponding to the window position: For each sliding window (row index r, column index c), its texture feature vector... Contrast eigenvectors and element eigenvectors Concatenate them into a total feature vector :

[0102] ;

[0103] The feature vector and the window's coordinate position The feature vector set is formed by storing the window index (r, c) in association. , where K is the total number of sliding windows. This set of feature vectors will serve as the input to the contamination identification module in step 104.

[0104] After completing step 103, a set of feature vectors corresponding to each sliding window position is obtained. Based on this, step 104 further inputs these feature vectors into a pre-trained classification model to identify suspected contaminated areas and determine the contamination category and confidence level. This specifically includes the following sub-steps:

[0105] 104-1, such as Figure 6 As shown, the feature vectors corresponding to each sliding window are input into a pre-trained classification model, which outputs the pollution category prediction result for each sliding window. The pre-trained classification model uses the random forest algorithm. The random forest consists of T=100 decision trees, and the training samples for each decision tree are extracted from the training set using a bootstrap sampling method. At each node split, m=3 features are randomly selected for optimal splitting. For any sliding window feature vector... The data is fed into a random forest model, and each decision tree outputs a pollution category prediction. Where 0 represents no pollution, 1 represents carbon pollution, 2 represents metal pollution, and 3 represents particulate pollution. The final prediction results of the random forest. Let be the mode of the predictions from the T decision trees. Simultaneously, record the confidence level p of this window, which represents the proportion of each predicted category across the T trees.

[0106] ;

[0107] in, This is an indicator function; it takes a value of 1 when the condition is met, and 0 otherwise. This yields the pollution category prediction result for each sliding window. and confidence level , where k is the window index.

[0108] 104-2. Based on the pollution category prediction results, adjacent sliding windows with the same pollution category are merged to obtain at least one connected region: using the row index r and column index c of the sliding window as coordinates, a label matrix L with the same size as the sliding window grid is constructed, where the initial value of each element is the predicted category of the window. The condition for two windows to be adjacent is defined as follows: the two windows are 4-connected (up, down, left, and right) in the grid. All windows of non-pollution-free categories (category values ​​1, 2, and 3) are traversed, and adjacent windows with the same category value are marked as connected components using a two-pass scanning algorithm. Each connected component is assigned a unique region identifier. Record the set of all window indices contained in this connected component. .

[0109] 104-3. Identify connected regions as suspected contaminated areas: For each connected region Calculate its bounding box in the scanning electron microscope image coordinate system. Let the set of coordinates of all window center points within this connected region be . Taking the smallest bounding rectangle as the boundary of the connected region, we obtain a set of polygon vertex coordinates. The connected region and its boundaries are designated as a suspected contaminated area, denoted as […]. .

[0110] 104-4. Obtain the pollution category prediction results and confidence values ​​for each sliding window constituting the suspected contaminated area: For each suspected contaminated area Get the set of window indices it contains. Extract the pollution category prediction results for each window. and confidence level .

[0111] 104-5. Count the frequency of each category in all sliding windows, and take the pollution category with the highest frequency as the pollution category of the suspected polluted area: For suspected polluted areas Count the frequency of the categories of its internal windows:

[0112] ;

[0113] Then the pollution category of the area Determined by the maximum frequency:

[0114] ;

[0115] If there are cases where frequencies are tied, the category with the higher total confidence score is selected.

[0116] 104-6. Calculate the statistic based on the confidence values ​​of all sliding windows, and use the statistic as the confidence level for the suspected contaminated area: For the suspected contaminated area... Calculate the confidence values ​​of all windows within it. The arithmetic mean of the values ​​is used as the confidence level for the region. :

[0117] ;

[0118] in, This represents the total number of windows contained within the region. This average value reflects the reliability of the overall region identification results.

[0119] Finally, each suspected contaminated area will be... Pollution categories Location information (polygon vertex coordinates) and confidence level Send it to step 105 together.

[0120] After completing step 104, the pollution category, location information, and confidence level of each suspected contaminated area are obtained. Based on this, step 105 further outputs these identification results and writes them back to the case library for subsequent use, specifically including the following sub-steps:

[0121] 105-1, such as Figure 7 As shown, the boundary coordinates or contour vertex coordinates of the suspected contamination area in the coordinate system of the scanning electron microscope image are used as location information: for each suspected contamination area The bounding box boundary obtained in step 104-3 is taken, which is the coordinate of the vertex of the smallest bounding rectangle in the scanning electron microscope image coordinate system. The vertex coordinates are arranged in clockwise order and denoted as:

[0122] ;

[0123] For a more refined outline representation, the vertex coordinates of the convex hull polygon formed by the center points of all sliding windows within the connected domain in step 104-2 can be used.

[0124] 105-2. Store pollution category, location information, and confidence level as a structured identification record: Assemble information for each suspected contaminated area into a structured identification record. The record format uses a key-value pair set and must contain at least the following fields: Area ID: (Integer); Pollution category: (Integer values ​​1, 2, and 3 correspond to carbon pollution, metal pollution, and particulate pollution, respectively); Location information: Boundary coordinate sequence (length 4 or more coordinate points); Confidence level: (A floating-point number between 0 and 1); Timestamp: The time stamp of the current analysis. Structured identification records are stored as JSON format or relational database table records for subsequent reading and writing.

[0125] 105-3. Output structured identification records to display terminals, storage media, or downstream analysis systems: Output the structured identification records generated in step 105-2 to a display terminal in a visual manner, for example, by overlaying the bounding boxes and labels of contaminated areas onto a scanning electron microscope image and displaying the confidence level values. Simultaneously, output the raw structured data to storage media (such as local disks or network file systems) or send it to downstream analysis systems (such as automated defect classification systems or yield management systems) to generate analysis reports or trigger subsequent process adjustments.

[0126] 105-4. Write the structured recognition records, along with their corresponding scanning electron microscope (SEM) images, transmission electron microscope (TEM) images, and energy dispersive spectroscopy (EDS) data, into a case library for updating the training set of the classification model or for use in manual review tasks. Establish a case library, physically a relational database or a file system directory. For this analysis task, perform the following write operations: First, insert the structured recognition record generated in step 105-2 as a metadata record into the data table of the case library. Second, copy the original multimodal data related to this analysis to the storage location of the case library, including: original SEM image files; original TEM image files; original EDS data files (containing spectral lines and spatial coordinate information); and the feature vector set generated in step 103 (optionally saved). Establish an association index between the metadata records and the original data files to ensure that the corresponding original data can be located based on the record ID.

[0127] The data accumulated in the case library serves two purposes: updating the training set of the classification model: regularly extracting correctly labeled data that has been manually verified from the case library and adding it to the training set of the random forest model in step 104-1 to retrain the model and improve recognition accuracy. Providing data for manual verification tasks: when the confidence level of the automatic recognition result is lower than a preset threshold (e.g., 0.7) or when the user needs verification, the corresponding original multimodal data and intermediate results are retrieved from the case library, and manually verified, corrected, or confirmed in the graphical interface. The corrected labeled data is then written back to the case library.

[0128] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An automatic identification method for interface contamination in advanced manufacturing processes, characterized in that, Includes the following steps: Acquire scanning electron microscope images, transmission electron microscope images, and energy dispersive spectroscopy data of the target interface area; Spatial alignment is performed on the scanning electron microscope image, the transmission electron microscope image, and the energy dispersive spectroscopy data to obtain multimodal aligned data in a unified coordinate system; Extract texture features, contrast features, and element features from the multimodal alignment data; The texture features, contrast features, and element features are input into a pre-trained classification model to identify suspected contamination areas in the target interface region and determine the contamination category and confidence level of the suspected contamination areas. Output the pollution category, the location information of the suspected polluted area, and the confidence level, and write the identification results back to the case library for subsequent model training or verification.

2. The method for automatic identification of interface contamination in advanced processes according to claim 1, characterized in that, The steps of acquiring scanning electron microscope images, transmission electron microscope images, and energy dispersive spectroscopy data of the target interface region specifically include: At least three shared marker points are prepared in the target interface region, and the shared marker points can be imaged and identified under both scanning electron microscopy and transmission electron microscopy. Scanning electron microscope images, transmission electron microscope images, and energy spectrum surface scan data or energy spectrum spot scan data covering the target interface region are acquired respectively. Record the spatial coordinate information of the scanning electron microscope image, the transmission electron microscope image, and the energy dispersive spectroscopy data in the image coordinate system.

3. The method for automatic identification of interface contamination in advanced processes according to claim 1, characterized in that, The steps for obtaining multimodal aligned data in a unified coordinate system specifically include: Extract the pixel coordinates of at least three shared marker points from the scanning electron microscope image and the transmission electron microscope image, respectively; Calculate the spatial transformation matrix based on the pixel coordinate correspondence of the shared marker points in the scanning electron microscope image and the transmission electron microscope image; The transmission electron microscope image is transformed to the coordinate system of the scanning electron microscope image using the spatial transformation matrix to obtain an aligned transmission electron microscope image; The acquisition coordinates of the energy spectrum data are mapped to the coordinate system of the scanning electron microscope image through a calibration relationship to obtain aligned energy spectrum data; The scanning electron microscope image, the aligned transmission electron microscope image, and the aligned energy dispersive spectroscopy data are collectively used as the multimodal alignment data.

4. The method for automatic identification of interface contamination in advanced processes according to claim 1, characterized in that, The steps for extracting texture features, contrast features, and element features specifically include: Within the image space of the multimodal alignment data, a sliding window is used to traverse the target interface region; For each sliding window, texture and contrast features are extracted from the aligned transmission electron microscope image or the scanning electron microscope image, and elemental features are extracted from the aligned energy spectroscopy data. The texture features, contrast features, and element features extracted from each sliding window are associated and stored according to the window position to form a feature vector set corresponding to the window position.

5. The method for automatic identification of interface contamination in advanced processes according to claim 1, characterized in that, The step of identifying suspected contaminated areas in the target interface region specifically includes: The feature vectors corresponding to each sliding window are input into the pre-trained classification model, and the classification model outputs the pollution category prediction result for each sliding window. Based on the pollution category prediction results, adjacent sliding windows with the same pollution category are merged into connected components to obtain at least one connected region. The connected region is identified as the suspected contaminated region.

6. The method for automatic identification of interface contamination in advanced processes according to claim 1, characterized in that, The steps for determining the pollution category and confidence level of the suspected contaminated area specifically include: Obtain the pollution category prediction results and confidence values ​​for each sliding window constituting the suspected contaminated area; The frequency of each category in all sliding windows is counted, and the pollution category with the highest frequency is taken as the pollution category of the suspected polluted area. A statistic is calculated based on the confidence values ​​of all the sliding windows, and the statistic is used as the confidence level of the suspected contaminated area.

7. The method for automatic identification of interface contamination in advanced processes according to claim 1, characterized in that, The step of writing the recognition results back to the case library for subsequent model training or verification specifically includes: The boundary coordinates or contour vertex coordinates of the suspected contaminated area in the coordinate system of the scanning electron microscope image are used as the location information; The pollution category, the location information, and the confidence level are associated and stored as a structured identification record; The structured recognition record is output to a display terminal, storage medium, or downstream analysis system; The structured recognition records, along with the corresponding scanning electron microscope images, transmission electron microscope images, and energy dispersive spectroscopy data, are written into a case library for updating the training set of the classification model or for use in manual review tasks.

8. An automatic identification device for interface contamination in an advanced manufacturing process, characterized in that, An automatic interface contamination identification method for an advanced process according to any one of claims 1-7, the apparatus comprising: The data acquisition module is used to acquire scanning electron microscope images, transmission electron microscope images, and energy dispersive spectroscopy data of the target interface area, and send the acquired data to the spatial alignment module. The spatial alignment module, connected to the data acquisition module, is used to receive the scanning electron microscope image, the transmission electron microscope image, and the energy dispersive spectroscopy data, and perform spatial alignment operations on them to obtain multimodal aligned data in a unified coordinate system, and then send the multimodal aligned data to the feature extraction module. The feature extraction module, connected to the spatial alignment module, is used to receive the multimodal alignment data, extract texture features, contrast features, and element features from the multimodal alignment data, and then send the texture features, contrast features, and element features to the contamination identification module. The contamination identification module, connected to the feature extraction module, is used to receive the texture features, the contrast features, and the element features, and input them into a pre-trained classification model to identify suspected contamination areas in the target interface area, determine the contamination category and confidence level of the suspected contamination area, and then send the contamination category, the location information of the suspected contamination area, and the confidence level to the output write-back module. The output write-back module, connected to the pollution identification module, is used to receive the pollution category, the location information of the suspected pollution area, and the confidence level, output the pollution category, the location information of the suspected pollution area, and the confidence level, and write the identification results back to the case library for subsequent model training or verification.

Images