Defect analysis and process optimization method based on artificial intelligence multi-modal fusion

Abstract

The invention relates to a defect analysis and process optimization method based on artificial intelligence multi-mode fusion, and the method comprises the steps: respectively inputting virtual data of a chip or device design layout, an expected process flow and parameters, and an expected electrical performance curve into an image encoder, a process encoder, and an electrical encoder, extracting a corresponding feature vector; performing cross-modal information interaction on the corresponding feature vectors according to a cross attention fusion mechanism; determining a representation image, a defect category, a root cause category and a process parameter correction suggestion; and based on a manufacturing execution system (MES), an electronic design automation (EDA) tool or an equipment interface, process parameters are corrected, and a closed-loop control result from defect analysis to process optimization is obtained. Therefore, the problems of single data dimension, mutual coupling of various defects, influence of background noise, limited machine identification accuracy, overlarge workload of artificial expert judgment and low efficiency are solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence multimodal fusion technology, and in particular to a defect analysis and process optimization method based on artificial intelligence multimodal fusion. Background Technology

[0002] As semiconductor manufacturing process nodes continue to evolve, Moore's Law is gradually approaching its limits. More and more device structures and packaging integration methods are being developed (FinFET, GAA, 3D NAND, heterogeneous integration, etc.). Chip systems are becoming increasingly complex, and the impact of defects on product performance and yield is significantly increasing.

[0003] Traditional defect analysis primarily relies on microscopic characterization techniques, commonly including SEM (Scanning Electron Microscopy), TEM (Transmission Electron Microscopy), and AFM (Atomic Force Microscopy). These methods allow for direct observation of defects on the wafer surface or cross-section (such as cracks, voids, impurity particles, and interface roughness). However, these technologies suffer from limitations due to their singular data dimensions, the interconnectedness of various defects, and the influence of background noise. Machine recognition accuracy is limited, while expert judgment is too labor-intensive and inefficient.

[0004] The ideas closest to this invention currently fall into the following categories: using deep learning to detect and classify defects in inspection images such as SEM, with some systems able to infer potential defect source processes by learning image patterns; using deep learning networks to automatically classify defects on wafer maps to assist yield engineers in making final decisions; and using process sequence modeling to predict wafer defect rates and locate possible process root causes.

[0005] Some existing technologies rely solely on microscopic images (such as SEM and TEM), which allow direct observation of defect morphology. However, defect type often requires human experience to determine, and the causal chain between defects and other manifestations is missing, making it difficult to pinpoint the true source of defects and resulting in low analysis efficiency. While some existing technologies incorporate process logs and parameters to help locate the root cause of defects, the complexity and tight coupling of process parameters, along with the intricate interactions between different steps, mean that traditional statistical methods can only identify correlations and cannot distinguish the inherent causal relationships. This can easily lead to "misjudging the root cause," resulting in ineffective and excessive process adjustments and wasting human and material resources. Summary of the Invention

[0006] This application provides a defect analysis and process optimization method based on artificial intelligence multimodal fusion to solve the problems of fragmented multimodal information, reliance on human experience for defect diagnosis, and lack of automated process optimization in the prior art.

[0007] The first aspect of this application provides a defect analysis and process optimization method based on artificial intelligence multimodal fusion, comprising the following steps: acquiring the chip or device design layout, expected process flow and parameters, and expected electrical performance curve of the target semiconductor device in the inference stage; inputting the virtual data of the chip or device design layout, the expected process flow and parameters, and the expected electrical performance curve into an image encoder, a process encoder, and an electrical encoder respectively to extract corresponding feature vectors; performing cross-modal information interaction on the corresponding feature vectors according to a cross-attention fusion mechanism to capture the correspondence between defect morphology, electrical effects, and process conditions; generating a characterization image of the correspondence through a diffusion model, and quantifying the responsibility of the process parameters based on the characterization image to obtain fused features; determining the characterization image, defect category, root cause category, and process parameter correction suggestions based on the fused features; and, if the analysis results meet preset confidence conditions, issuing the characterization image, the defect category, the root cause category, and the process parameter correction suggestions using a Manufacturing Execution System (MES), Electronic Design Automation (EDA) tools, or device interfaces to correct the process parameters, thereby obtaining a closed-loop control result from defect analysis to process optimization.

[0008] Optionally, in one embodiment of this application, before acquiring the chip or device design layout, expected process flow and parameters, and expected electrical performance curve of the target semiconductor device in the inference stage, the method further includes: acquiring the characterization image, electrical curve, and process parameters of the target semiconductor device in the training stage; preprocessing the characterization image, electrical curve, and process parameters of the training stage to obtain preprocessed data; inputting the preprocessed data into the image encoder, the electrical encoder, and the process encoder to extract the corresponding feature vectors; and merging the corresponding features according to the cross-attention fusion mechanism. Vector interaction modeling is used to capture the correspondence between the defect morphology, electrical effects, and process conditions. A counterfactual attribution mechanism is used to simulate the changes in the model prediction results under preset conditions, so as to quantify the contribution of the process parameters to the generation of defects based on the changes. Based on the correspondence and the contribution, the prediction results are corrected using a preset electrical surrogate model or a preset thermodynamic surrogate model to generate constraint results that satisfy the physical laws of the target semiconductor device. Based on the constraint results, the loss function is backpropagated to the preset model through gradient to jointly optimize the multimodal encoder, fusion layer, and prediction head to generate process optimization results.

[0009] Optionally, in one embodiment of this application, obtaining the characterization image, electrical curve, and process parameters of the target semiconductor device during the training phase includes: obtaining the characterization image from a scanning electron microscope or a transmission electron microscope; obtaining the electrical curve generated by an electrical testing device; and obtaining the process parameters including deposition rate, etching power, annealing temperature, and time.

[0010] Optionally, in one embodiment of this application, the step of preprocessing the representation images, electrical curves, and process parameters from the training phase to obtain preprocessed data includes: performing noise removal, block segmentation, and data augmentation on the representation images to obtain processed images; unifying the data of the electrical curves to the same dimension, and extracting key electrical features including threshold voltage, leakage current, and subthreshold slope under the same dimension; filtering, windowing, and standardizing the process parameters to obtain processed process parameters, and temporally corresponding the processed process parameters with the processed images and the key electrical features to obtain the preprocessed data.

[0011] Optionally, in one embodiment of this application, the step of backpropagating the loss function to a preset model via gradient to jointly optimize the multimodal encoder, the fusion layer, and the prediction head to generate the process optimization result includes: obtaining the loss function comprising classification loss, root cause prediction loss, process correction regression loss, physical consistency loss, and counterfactual attribution loss; and backpropagating the loss function to the preset model via gradient to jointly optimize the multimodal encoder, the fusion layer, and the prediction head to generate the generated process optimization result.

[0012] A second aspect of this application provides a defect analysis and process optimization apparatus based on artificial intelligence multimodal fusion, comprising: an acquisition module for acquiring the chip or device design layout, expected process flow and parameters, and expected electrical performance curve of a target semiconductor device during the inference stage; an extraction module for inputting virtual data of the chip or device design layout, the expected process flow and parameters, and the expected electrical performance curve into an image encoder, a process encoder, and an electrical encoder, respectively, to extract corresponding feature vectors; and a quantization module for performing cross-modal information interaction on the corresponding feature vectors according to a cross-attention fusion mechanism to capture defect morphology and electrical effects. The system establishes a correspondence between process conditions and parameters, generates a characterization image based on this correspondence using a diffusion model, and quantifies the responsibility of the process parameters according to the characterization image to obtain fusion features. A determination module is used to determine the characterization image, defect category, root cause category, and process parameter correction suggestions based on the fusion features. An optimization module is used to, when the analysis results meet preset confidence conditions, distribute the characterization image, defect category, root cause category, and process parameter correction suggestions using a Manufacturing Execution System (MES), Electronic Design Automation (EDA) tools, or equipment interfaces to correct the process parameters, thereby obtaining a closed-loop control result from defect analysis to process optimization.

[0013] Optionally, in one embodiment of this application, it further includes: a data acquisition module, used to acquire the characterization image, electrical curve, and process parameters of the target semiconductor device during the training phase before acquiring the chip or device design layout, expected process flow and parameters, and expected electrical performance curve of the target semiconductor device during the inference phase; a preprocessing module, used to preprocess the characterization image, electrical curve, and process parameters of the training phase respectively to obtain preprocessed data; a feature extraction module, used to input the preprocessed data into the image encoder, the electrical encoder, and the process encoder respectively to extract the corresponding feature vectors; and a multimodal fusion and causal inference module, based on the cross-attention fusion machine. The system interactively models the corresponding feature vectors to capture the correspondence between the defect morphology, electrical effects, and process conditions, and uses a counterfactual attribution mechanism to simulate the changes in the model prediction results under preset conditions, so as to quantify the contribution of the process parameters to defect generation based on the changes; a physical consistency constraint module is used to correct the prediction results based on the correspondence and the contribution, using a preset electrical proxy model or a preset thermodynamic proxy model, so as to generate constraint results that satisfy the physical laws of the target semiconductor device; a closed-loop execution module is used to backpropagate the loss function to the preset model through gradient based on the constraint results, so as to jointly optimize the multimodal encoder, fusion layer, and prediction head, and generate process optimization results.

[0014] Optionally, in one embodiment of this application, the data acquisition module includes: an image acquisition unit for acquiring the characterization image from a scanning electron microscope or a transmission electron microscope; a curve acquisition unit for acquiring the electrical curve generated by an electrical testing device; and a data acquisition unit for acquiring the process parameters including deposition rate, etching power, annealing temperature, and time.

[0015] Optionally, in one embodiment of this application, the preprocessing module includes: a processing unit, configured to perform noise removal, block segmentation, and data augmentation processing on the characterization image to obtain a processed image; an extraction unit, configured to unify the data of the electrical curve to the same dimension, and extract key electrical features including threshold voltage, leakage current, and subthreshold slope under the same dimension; and a generation unit, configured to perform filtering, windowing, and standardization processing on the process parameters to obtain processed process parameters, and to correlate the processed process parameters with the processed image and the key electrical features in time sequence to obtain the preprocessed data.

[0016] Optionally, in one embodiment of this application, the closed-loop execution module includes: a data acquisition unit, used to acquire the loss function including classification loss, root cause prediction loss, process correction regression loss, physical consistency loss, and counterfactual attribution loss; and a joint optimization unit, used to backpropagate the loss function to the preset model via gradient to jointly optimize the multimodal encoder, the fusion layer, and the prediction head to generate the generation process optimization result.

[0017] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement the defect analysis and process optimization method based on artificial intelligence multimodal fusion as described in the above embodiments.

[0018] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for defect analysis and process optimization based on artificial intelligence multimodal fusion.

[0019] A fifth aspect of this application provides a computer program product that stores a computer program that, when executed by a processor, implements the above-described method for defect analysis and process optimization based on artificial intelligence multimodal fusion.

[0020] This application's embodiments collect data such as chip / device design layout, expected process flow and parameters, and expected electrical performance curves. An encoder is used to extract images, process sequences, and electrical features. Interactive modeling and unified characterization are performed at a multimodal fusion layer to achieve rapid defect identification, process source localization, and specific corrective measures. This solves the problems of existing image-based defect analysis, which only provides morphological information but cannot reveal the causal relationship between defects and processes or electrical performance; existing statistical analysis based on process logs, which can only find correlations but cannot provide accurate causal reasoning, easily leading to misjudgments and ineffective corrections; existing electrical curve-based analysis methods can reflect performance anomalies but cannot correspond to specific defect types and process stages; and existing multimodal solutions fuse at most two types of data, failing to form a closed-loop chain of "defect image—electrical effect—process conditions." Furthermore, existing technologies typically require tape-out to obtain corresponding measured characterization images and data before defect analysis and other related steps can be performed.

[0021] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0022] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a defect analysis and process optimization method based on artificial intelligence multimodal fusion provided according to an embodiment of this application; Figure 2 This is a flowchart of the inference stage of a defect analysis and process optimization method based on artificial intelligence multimodal fusion according to an embodiment of this application; Figure 3 This is a flowchart of the training phase of a defect analysis and process optimization method based on artificial intelligence multimodal fusion according to an embodiment of this application; Figure 4 This is a schematic diagram of a defect analysis and process optimization device based on artificial intelligence multimodal fusion provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0023] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0024] The following describes an embodiment of the defect analysis and process optimization method based on artificial intelligence multimodal fusion, with reference to the accompanying drawings. Addressing the problems mentioned in the background section regarding fragmented multimodal information, reliance on human experience for defect diagnosis, and lack of automated process optimization in existing technologies, this application provides a defect analysis and process optimization method based on artificial intelligence multimodal fusion. In this method, data such as chip / device design layout, expected process flow and parameters, and expected electrical performance curves are collected. An encoder is used to extract image, process sequence, and electrical features. Interactive modeling and unified representation are performed at the multimodal fusion layer to achieve rapid defect identification, process source localization, and specific corrective measures, thus solving the problems of fragmented multimodal information, reliance on human experience for defect diagnosis, and lack of automated process optimization in existing technologies. More importantly, this method generates corresponding representation images based on virtual data (design data / expected data) during the inference stage, performs defect judgment, root cause identification, and provides corresponding process correction schemes. This virtual representation approach can effectively contain potential risks before the tape-out process, significantly reducing actual manufacturing costs and providing significant guidance for process engineers and design engineers. This solution is applicable to advanced semiconductor device manufacturing processes, including CMOS, 3D NAND, TSV interconnects, and heterogeneous integration.

[0025] Specifically, Figure 1 This is a flowchart illustrating a defect analysis and process optimization method based on artificial intelligence multimodal fusion, provided in an embodiment of this application.

[0026] like Figure 1 As shown, the defect analysis and process optimization method based on artificial intelligence multimodal fusion includes the following steps: In step S101, the chip or device design layout, expected process flow and parameters, and expected electrical performance curve of the target semiconductor device in the inference stage are obtained.

[0027] It is understood that the inference stage in the embodiments of this application includes a generation task, which outputs a representation image.

[0028] In actual implementation, such as Figure 2 As shown, in the inference phase of this application, a multimodal artificial intelligence model optimized during the training phase can be used to process multiple types (three or more types) of data, including virtual data such as chip / device design layout, expected process flow and parameters, and expected electrical performance curves. After matching and alignment, the data is sent to the subsequent processing module. Compared to single-modality models, this application utilizes multimodal (trimodal) joint modeling to reduce information loss and ensure more comprehensive information.

[0029] In the inference stage, this application, for the first time, uses virtual data (including chip / device design layout, expected process flow and parameters, expected electrical performance curves, etc.) as a unified input. Finally, a diffusion model is used to generate a characterization image, while simultaneously performing defect judgment, root cause identification, and process correction. This allows potential risks to be identified in advance before the formal tape-out process, providing specific optimization suggestions and significantly reducing manufacturing costs.

[0030] In step S102, the virtual data of the chip or device design layout, expected process flow and parameters, and expected electrical performance curve are input into the image encoder, process encoder, and electrical encoder, respectively, to extract the corresponding feature vectors.

[0031] In actual implementation, the data in this embodiment undergoes feature extraction via encoders: an image encoder extracts spatial topography features of the layout, a process encoder extracts process timing features, and an electrical encoder extracts curve features. Since this stage uses parameters optimized during the training phase, fast and stable feature representation is guaranteed. The output is a three-way feature vector, which serves as input for subsequent fusion analysis.

[0032] In step S103, the corresponding feature vectors are subjected to cross-modal information interaction according to the cross-attention fusion mechanism to capture the correspondence between defect morphology, electrical effects and process conditions. The correspondence is used to generate a characterization image through a diffusion model, and the responsibility of process parameters is quantified according to the characterization image to obtain the fused features.

[0033] In the fusion inference module, the three feature vectors interact across modalities through a cross-attention fusion mechanism, capturing the correspondence between "defect morphology—electrical effects—process conditions" and generating a representation image through a diffusion model. Simultaneously, the causal analysis and counterfactual attribution module can directly infer the possible process steps from which the defect originates and quantify the degree of responsibility for each process step. The output of this module is a fused feature representation and a causal chain.

[0034] This application embodiment can perform multimodal coding and Cross-Attention fusion, and designs three types of encoders: image, electrical, and process. It also achieves cross-modal information interaction through the Cross-Attention mechanism, explicitly capturing "defects". Electrical anomaly The relationship between process parameters.

[0035] In step S104, based on the fusion features, the characterization image, defect category, root cause category, and process parameter correction suggestions are determined.

[0036] In this embodiment, the system can output multiple results based on fusion features: characterization image, defect category, root cause category, and process parameter correction suggestions. When confirmed by the engineer or judged with high confidence by the system, the prediction results are sent to the closed-loop execution module. Through the interface with MES (Manufacturing Execution System), EDA tools or equipment, process correction instructions are automatically or semi-automatically issued to realize real-time adjustment of the manufacturing process, thereby improving yield and stability.

[0037] The embodiments of this application can perform multi-task joint output. Under physical consistency constraints, the model can not only generate characterization images, but also output defect categories, defect root causes, process parameter correction amounts (numerical regression), prediction confidence intervals, and interpretation reports.

[0038] In step S105, if the analysis results meet the preset confidence conditions, the system uses the Manufacturing Execution System (MES), Electronic Design Automation (EDA) tools, or equipment interfaces to issue characterization images, defect categories, root cause categories, and process parameter correction suggestions to correct the process parameters and obtain closed-loop control results from defect analysis to process optimization.

[0039] Specifically, the embodiments of this application can achieve closed-loop optimization and human-machine collaboration. The analysis results can be sent through MES / EDA / equipment interfaces to achieve automatic or semi-automatic process parameter correction; at the same time, interpretable reports are generated for engineers to confirm, realizing human-machine collaborative closed-loop optimization and obtaining closed-loop control results from defect analysis to process optimization. This application can reduce repeated manual trial and error, accelerate the iteration cycle of defect analysis and process optimization, and ensure higher efficiency.

[0040] This application goes beyond testing; it also provides suggestions for process parameter-level corrections, enabling closed-loop optimization and ensuring more accurate corrections.

[0041] Optionally, in one embodiment of this application, before acquiring the chip or device design layout, expected process flow and parameters, and expected electrical performance curve of the target semiconductor device in the inference stage, the method further includes: acquiring the characterization image, electrical curve, and process parameters of the target semiconductor device in the training stage; preprocessing the characterization image, electrical curve, and process parameters in the training stage to obtain preprocessed data; inputting the preprocessed data into the image encoder, electrical encoder, and process encoder to extract the corresponding feature vectors; interactively modeling the corresponding feature vectors according to the cross-attention fusion mechanism to capture the correspondence between defect morphology, electrical effects, and process conditions, and using the counterfactual attribution mechanism to simulate the change of the model prediction result under preset conditions to quantify the contribution of process parameters to defect generation based on the change; correcting the prediction result using a preset electrical proxy model or a preset thermodynamic proxy model based on the correspondence and contribution to generate constraint results that satisfy the physical laws of the target semiconductor device; and backpropagating the loss function to the preset model through gradient based on the constraint results to jointly optimize the multimodal encoder, fusion layer, and prediction head to generate process optimization results.

[0042] It is understood that the preset electrical proxy model or preset thermodynamic proxy model in the embodiments of this application can be a differentiable electrical proxy model or thermodynamic proxy model.

[0043] In actual implementation, such as Figure 3 As shown, embodiments of this application can acquire characterization images, electrical curves, and process parameters of a target semiconductor device during the training phase; preprocess the characterization images, electrical curves, and process parameters during the training phase to obtain preprocessed data; and input the preprocessed data into an image encoder, an electrical encoder, and a process encoder to extract the corresponding feature vectors.

[0044] In the feature fusion and causal inference module, the three features interact and are modeled through a cross-attention mechanism to learn the "defect image". Electrical deviation The module identifies the inherent relationships between process conditions. It also introduces a counterfactual attribution mechanism to simulate changes in model predictions under conditions of removing or altering a process parameter, thereby quantifying the contribution of each process step to defect generation. The unified features output by this module contain both defect pattern information and causal attribution signals.

[0045] Furthermore, this invention introduces a physical consistency constraint module, utilizing a differentiable electrical or thermodynamic surrogate model to correct the prediction results, ensuring that the model output conforms to the physical laws of semiconductors and improving the interpretability and credibility of the model. This application's embodiments add causal inference and counterfactual analysis within the model, simulating the impact of changes in process conditions on defect probability and outputting the responsibility of each process step, elevating the inference from "correlation" to "causality." The introduction of a differentiable electrical / thermodynamic surrogate model constrains the model output, ensuring the prediction results meet the physical laws of semiconductor devices, improving the credibility and interpretability of the results. Physical consistency constraints and interpretability report outputs make the results engineering-verifiable, guaranteeing greater credibility. Causal / counterfactual inference can be achieved through partial trajectory regression and virtual path simulation, while physical consistency constraints can be achieved through a differentiable electrical / thermodynamic surrogate model. Through counterfactual inference, this application can directly answer "which process step caused the defect," ensuring clearer causality.

[0046] In the multi-task output and loss calculation module, the model output includes defect type classification results, defect root cause process step prediction, and regression values ​​of specific process correction amounts, and provides prediction confidence intervals or Top-k suggestions.

[0047] This application embodiment can fuse multimodal (three-modal or more) inputs. During the training phase, for the first time, characterization images (SEM / TEM / AFM, etc.), process sequence steps and parameters (deposition rate / etching time / annealing temperature, etc.), electrical curves (IV / CV, etc.) are used as unified inputs to ensure that the entire chain of information on defect morphology, electrical effects and process conditions is utilized simultaneously.

[0048] Optionally, in one embodiment of this application, obtaining characterization images, electrical curves, and process parameters of the target semiconductor device during the training phase includes: obtaining characterization images from a scanning electron microscope or a transmission electron microscope; obtaining electrical curves generated by an electrical testing device; and obtaining process parameters including deposition rate, etching power, annealing temperature, and time.

[0049] In practical implementation, this application can begin with multimodal input data, including characterization images from SEM / TEM, electrical curves (IV, CV, etc.) from electrical testing equipment, and process parameters (deposition rate, etching power, annealing temperature and time, etc.) from the manufacturing execution system. These three types of data are matched and aligned using device ID, layout coordinates, and timestamps to ensure that the input data corresponds to the same device or process stage. Through unified feature and timing alignment, this application remains effective across different device and process environments, ensuring stronger generalization.

[0050] Optionally, in one embodiment of this application, the representation images, electrical curves, and process parameters from the training phase are preprocessed to obtain preprocessed data. This includes: performing noise removal, block segmentation, and data augmentation on the representation images to obtain processed images; unifying the electrical curve data to the same dimension, and extracting key electrical features including threshold voltage, leakage current, and subthreshold slope within the same dimension; filtering, windowing, and standardizing the process parameters to obtain processed process parameters, and temporally mapping the processed process parameters to the processed images and key electrical features to obtain preprocessed data.

[0051] It is understandable that image data preprocessing methods include, but are not limited to, denoising, registration, and defect synthesis augmentation; electrical curve feature extraction methods include, but are not limited to, threshold voltage, leakage current, and subthreshold slope; process parameters can be processed using windowing and timing alignment methods.

[0052] In the data preprocessing stage, this embodiment of the application can perform noise removal, block segmentation, and data augmentation on image data to improve the robustness of the model to defect morphology; electrical curve data is normalized and sampled to the same dimension, while key electrical features such as threshold voltage, leakage current, and subthreshold slope are extracted; process parameter sequences are filtered, windowed, and standardized to correspond temporally with the image and electrical data. The three types of preprocessed data are respectively sent to their respective feature extraction modules: image encoders (CNN, ViT, etc.) extract spatial and morphological features, electrical encoders (1D-CNN, Transformer, etc.) extract curve morphology features, and process encoders (temporal Transformer, RNN, etc.) extract evolutionary features between process steps.

[0053] Optionally, in one embodiment of this application, the loss function is backpropagated to a preset model via gradient to jointly optimize the multimodal encoder, fusion layer, and prediction head, generating process optimization results. This includes: obtaining a loss function that includes classification loss, root cause prediction loss, process correction regression loss, physical consistency loss, and counterfactual attribution loss; and backpropagating the loss function to a preset model via gradient to jointly optimize the multimodal encoder, fusion layer, and prediction head, generating process optimization results.

[0054] Specifically, the loss function in this application embodiment is a weighted combination of classification loss, root cause prediction loss, process correction regression loss, physical consistency loss, and counterfactual attribution loss.

[0055] The loss signal is fed back into the model via gradient feedback, enabling joint optimization of the multimodal encoder, fusion layer, and prediction head. This continuously updates the parameters, improving the model's accuracy and generalization ability in defect identification, root cause localization, and process modification suggestions. The entire training phase thus forms a closed loop: from raw data acquisition → preprocessing → feature extraction → fusion and causal modeling → physical constraints → multi-task prediction → loss optimization → parameter update.

[0056] It should be noted that: 1. Single-modal alternative: Image-based defect detection: Convolutional neural networks (CNNs) or visual Transformers can be used to classify defects in SEM / TEM images. Advantages: Simple to implement, and has been extensively studied. Disadvantages: Cannot provide causal relationships between defects and process conditions or electrical properties, making it difficult to offer process modification suggestions.

[0057] Anomaly detection based solely on electrical curves: Traditional machine learning or deep learning can be used to perform pattern recognition on IV / CV curves to determine if a device has defects. Advantages: Electrical testing is fast and readily available. Disadvantages: It is difficult to distinguish defect types and cannot trace back to specific process stages.

[0058] Statistical analysis based solely on process sequence parameters (logs, etc.): This method uses statistical learning or regression to find correlations between process parameters and yield. Advantages: Easy to integrate into existing process systems. Disadvantages: Only detects correlations, lacks causal explanations, and correction suggestions are not precise enough.

[0059] 2. Alternative technical implementation methods Even in multimodal scenarios, different technical approaches may be adopted: Fusion method replacement: In addition to Cross-Attention fusion, multimodal contrastive learning, simple feature concatenation, or co-training methods can be used. Causal reasoning replacement: Bayesian networks, structural causal models (SCM), etc., can replace counterfactual reasoning modules. Physical constraint replacement: Predictions can be verified through rule-based process constraints, finite element simulation results, or empirical formulas, without necessarily using differentiable surrogate models.

[0060] The above alternative solutions can complete defect detection and analysis to a certain extent, but they generally have the following shortcomings: (1) Incomplete information dimensions and incomplete causal chain; (2) The output is limited to the level of "defect identification" and cannot generate executable process correction suggestions; (3) Lack of physical consistency verification and limited credibility of results; (4) Lack of human-machine collaboration and closed-loop execution mechanism; (5) Reliance on actual test data after tape-out, resulting in huge costs.

[0061] This application provides a more complete, reliable, and practical solution through multimodal (trimodal and above) fusion during the training phase, causal / counterfactual reasoning, physical consistency constraints, closed-loop execution mechanisms, and virtual data fusion during the inference phase.

[0062] According to the defect analysis and process optimization method based on artificial intelligence multimodal fusion proposed in the embodiments of this application, the following steps are taken: 1. In the training phase, joint modeling of multimodal (three or more modal) data is achieved: characterization images (SEM / TEM / AFM, etc.), process sequence steps and parameters (deposition rate / etching time / annealing temperature, etc.), and electrical curves (IV / CV, etc.) are simultaneously used as inputs to overcome the single nature of existing methods. 2. Establishing a defect causal chain: through a multimodal artificial intelligence model, a causal relationship is established between "defect image - process root cause - electrical effect" to achieve automated defect identification and source localization. 3. Outputting executable process correction suggestions: based on interpretable AI and causal reasoning, specific process parameter adjustment directions and numerical ranges are given (such as reducing the annealing temperature by 10–20℃), rather than simply stopping at the defect detection stage. 4. Forming a closed-loop optimization system: the defect analysis results are fed back to the process links to help achieve closed-loop control for yield improvement. 5. In the inference stage, the input data for multimodal processing are all virtual data (including chip / device design layout, expected process flow and parameters, expected electrical performance curves, etc.). Based on the virtual representation data, defect judgment, root cause identification, and process correction can be performed. Potential risks can be identified in advance before the formal tape-out task, and specific optimization suggestions can be given, significantly reducing manufacturing costs.

[0063] The technical problems addressed in this application are as follows: 1. In the training phase, effectively integrate multiple types (three or more) of data, such as microscopic images, process parameters, and electrical curves, overcoming the limitations of existing technologies that rely on only a single modality; 2. Through causal reasoning and multimodal deep learning, establish a mapping relationship between defect types, process sources, and electrical effects to avoid root cause misjudgment; 3. After defect analysis, further output quantitative and actionable process correction suggestions, rather than merely remaining at the detection and diagnosis level; 4. Ensure the model's generalization and interpretability under different devices and process environments, facilitating engineers' understanding and verification, and improving its practical applicability; 5. In the inference phase, use virtual representation data for defect judgment, root cause identification, and process correction, identifying potential risks in advance, providing specific optimization suggestions, and significantly reducing manufacturing costs.

[0064] Next, referring to the accompanying drawings, a defect analysis and process optimization apparatus based on artificial intelligence multimodal fusion proposed according to an embodiment of this application is described.

[0065] Figure 4 This is a schematic diagram of the structure of the defect analysis and process optimization device based on artificial intelligence multimodal fusion according to an embodiment of this application.

[0066] like Figure 4 As shown, the defect analysis and process optimization device 10 based on artificial intelligence multimodal fusion includes: an acquisition module 100, an extraction module 200, a quantification module 300, a determination module 400, and an optimization module 500.

[0067] Specifically, the acquisition module 100 is used to acquire the chip or device design layout, expected process flow and parameters, and expected electrical performance curve of the target semiconductor device during the inference stage.

[0068] The extraction module 200 is used to input virtual data of chip or device design layout, expected process flow and parameters, and expected electrical performance curves into the image encoder, process encoder and electrical encoder respectively, so as to extract the corresponding feature vectors.

[0069] The quantization module 300 is used to perform cross-modal information interaction on the corresponding feature vectors according to the cross-attention fusion mechanism to capture the correspondence between defect morphology, electrical effects and process conditions. The correspondence is used to generate a characterization image through a diffusion model, and the responsibility of process parameters is quantified according to the characterization image to obtain fused features.

[0070] The determination module 400 is used to determine the characterization image, defect category, root cause category, and process parameter correction suggestions based on the fused features.

[0071] The optimization module 500 is used to issue characterization images, defect categories, root cause categories, and process parameter correction suggestions through the Manufacturing Execution System (MES), Electronic Design Automation (EDA) tools, or equipment interfaces when the analysis results meet the preset confidence conditions, so as to correct the process parameters and obtain closed-loop control results from defect analysis to process optimization.

[0072] Optionally, in one embodiment of this application, the defect analysis and process optimization device 10 based on artificial intelligence multimodal fusion further includes: a data acquisition module, a preprocessing module, a feature extraction module, a multimodal fusion and causal inference module, a physical consistency constraint module, and a closed-loop execution module.

[0073] The data acquisition module is used to acquire characterization images, electrical curves, and process parameters of the target semiconductor device during the training phase, before acquiring the chip or device design layout, expected process flow and parameters, and expected electrical performance curves of the target semiconductor device during the inference phase.

[0074] The preprocessing module is used to preprocess the representation images, electrical curves, and process parameters from the training phase to obtain preprocessed data.

[0075] The feature extraction module is used to input the preprocessed data into the image encoder, electrical encoder and process encoder respectively to extract the corresponding feature vectors.

[0076] The multimodal fusion and causal inference module uses a cross-attention fusion mechanism to interactively model the corresponding feature vectors to capture the correspondence between defect morphology, electrical effects and process conditions. It also uses a counterfactual attribution mechanism to simulate the changes in model prediction results under preset conditions, so as to quantify the contribution of process parameters to defect generation based on the changes.

[0077] The physical consistency constraint module is used to correct the prediction results based on the correspondence and contribution, using a preset electrical proxy model or a preset thermodynamic proxy model, so as to generate constraint results that satisfy the physical laws of the target semiconductor device.

[0078] The closed-loop execution module is used to backpropagate the loss function to the preset model through gradient based on the constraint results, so as to jointly optimize the multimodal encoder, fusion layer and prediction head and generate process optimization results.

[0079] Optionally, in one embodiment of this application, the data acquisition module includes an image acquisition unit and a data acquisition unit.

[0080] The image acquisition unit is used to acquire characterization images from a scanning electron microscope or a transmission electron microscope; the curve acquisition unit acquires electrical curves generated by electrical testing equipment.

[0081] The data acquisition unit is used to acquire process parameters including deposition rate, etching power, annealing temperature, and time.

[0082] Optionally, in one embodiment of this application, the preprocessing module includes a processing unit, an extraction unit, and a generation unit.

[0083] The processing unit is used to perform noise removal, block segmentation, and data augmentation on the characterization image to obtain the processed image.

[0084] The extraction unit is used to unify the data of the electrical curves to the same dimension, and extract key electrical features including threshold voltage, leakage current and subthreshold slope under the same dimension.

[0085] The generation unit is used to filter, window, and standardize the process parameters to obtain the processed process parameters, and to correlate the processed process parameters with the processed images and key electrical features in time sequence to obtain the preprocessed data.

[0086] Optionally, in one embodiment of this application, the closed-loop execution module includes a data acquisition unit and a joint optimization unit.

[0087] The data acquisition unit is used to acquire a loss function that includes classification loss, root cause prediction loss, process correction regression loss, physical consistency loss, and counterfactual attribution loss.

[0088] The joint optimization unit is used to backpropagate the loss function to the preset model through gradients to jointly optimize the multimodal encoder, fusion layer and prediction head, and generate optimized results for the generation process.

[0089] It should be noted that the foregoing explanation of the embodiment of the defect analysis and process optimization method based on artificial intelligence multimodal fusion also applies to the defect analysis and process optimization device based on artificial intelligence multimodal fusion in this embodiment, and will not be repeated here.

[0090] The defect analysis and process optimization device based on artificial intelligence multimodal fusion proposed in this application ensures comprehensive information by using multimodal (three or more modal) inputs during the training phase, including images, process data, and electrical parameters. Cross-Attention fusion and a causal / counterfactual module are introduced to elevate the analysis from correlation to causality, enabling quantitative analysis of the impact of process parameters on defects. A physical consistency constraint module is added to ensure that the prediction results conform to the physical laws of the device, reducing "spurious correlations." Furthermore, this application generates characterization images and provides defect categories through virtual data input during the inference phase, before actual tape-out, and also provides root cause analysis and process parameter correction suggestions, directly leading to process improvement, significantly saving tape-out costs and nipping risks in the bud. The human-machine collaboration and closed-loop execution approach ensures that the AI ​​results are truly implemented and automatically fed back to the production system. This solves the problem of existing technologies lacking three-modal coherence, thus failing to guarantee the completeness and robustness of predictions. Because existing technologies only focus on correlation, they cannot answer questions such as "Will the defect disappear if the process conditions are changed?" Therefore, in modern semiconductor manufacturing, where defects are complex and processes are highly variable, traditional analysis methods inevitably lead to low analysis efficiency and high optimization costs. Furthermore, existing defect analysis techniques rely on post-fabrication measurement results, resulting in significant costs.

[0091] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0092] When the processor 502 executes the program, it implements the defect analysis and process optimization method based on artificial intelligence multimodal fusion provided in the above embodiments.

[0093] Furthermore, electronic devices also include: Communication interface 503 is used for communication between memory 501 and processor 502.

[0094] The memory 501 is used to store computer programs that can run on the processor 502.

[0095] Memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0096] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0097] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0098] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0099] This embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described defect analysis and process optimization method based on artificial intelligence multimodal fusion.

[0100] This application also provides a computer program product on which a computer program is stored. When the program is executed by a processor, it implements the above-mentioned defect analysis and process optimization method based on artificial intelligence multimodal fusion.

[0101] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0102] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0103] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0104] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0105] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0106] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0107] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0108] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A defect analysis and process optimization method based on artificial intelligence multi-modal fusion, characterized in that, The method comprises the following steps: obtaining a chip or device design layout of a target semiconductor device in an inference stage, an expected process flow and parameters, and an expected electrical performance curve; inputting virtual data of the chip or device design layout, the expected process flow and parameters, and the expected electrical performance curve into an image encoder, a process encoder, and an electrical encoder respectively to extract corresponding feature vectors; cross-modal information interaction of the corresponding feature vectors is performed according to a cross-attention fusion mechanism to capture a corresponding relationship between a defect morphology, an electrical effect, and a process condition, a representation image is generated through a diffusion model based on the corresponding relationship, and a responsibility degree of the process parameters is quantified according to the representation image to obtain fusion features; based on the fusion features, the representation image, a defect category, a root cause category, and a process parameter correction suggestion are determined; in a case where an analysis result meets a pre-set confidence condition, the representation image, the defect category, the root cause category, and the process parameter correction suggestion are issued through a manufacturing execution system (MES), an electronic design automation (EDA) tool, or a device interface to correct the process parameters, and a closed-loop control result from defect analysis to process optimization is obtained.

2. The method of claim 1, wherein, Before obtaining a chip or device design layout of a target semiconductor device in an inference stage, an expected process flow and parameters, and an expected electrical performance curve, the method further comprises: obtaining a representation image, an electrical curve, and process parameters of the target semiconductor device in a training stage; preprocessing the representation image, the electrical curve, and the process parameters in the training stage to obtain preprocessed data; inputting the preprocessed data into the image encoder, the electrical encoder, and the process encoder to extract the corresponding feature vectors; interaction modeling of the corresponding feature vectors is performed according to the cross-attention fusion mechanism to capture a corresponding relationship between a defect morphology, an electrical effect, and a process condition, and a change in a model prediction result under a pre-set condition is simulated through a counterfactual attribution mechanism to quantify a contribution degree of the process parameters to defects according to the change; based on the corresponding relationship and the contribution degree, a pre-set electrical proxy model or a pre-set thermodynamic proxy model is used to correct the prediction result to generate a constraint result meeting a physical law of the target semiconductor device; based on the constraint result, a loss function is back-propagated to a pre-set model through a gradient to jointly optimize a multi-modal encoder, a fusion layer, and a prediction head to generate a process optimization result.

3. The method of claim 2, wherein, The obtaining of the representation image, the electrical curve, and the process parameters of the target semiconductor device in the training stage comprises: obtaining the representation image from a scanning electron microscope or a transmission electron microscope; obtaining the electrical curve generated by an electrical test device; obtaining the process parameters including a deposition rate, an etching power, an annealing temperature, and a time.

4. The method of claim 2, wherein, The preprocessing of the representation image, the electrical curve, and the process parameters in the training stage to obtain preprocessed data comprises: performing noise removal, block segmentation, and data enhancement processing on the representation image to obtain a processed image; unify data of the electrical curve to the same dimension, and extract key electrical characteristics including threshold voltage, leakage current and sub-threshold slope under the same dimension; The process parameters are respectively screened, windowed and standardized to obtain processed process parameters, and the processed process parameters are corresponded in time sequence with the processed images and the key electrical characteristics to obtain the pre-processed data.

5. The method of claim 2, wherein, The loss function is back propagated to the preset model by gradient to jointly optimize the multi-modal encoder, the fusion layer and the prediction head to generate a process optimization result, including: Obtain the loss function including classification loss, root cause prediction loss, process correction regression loss, physical consistency loss and counterfactual attribution loss; The loss function is back propagated to the preset model by gradient to jointly optimize the multi-modal encoder, the fusion layer and the prediction head to generate the generated process optimization result.

6. An apparatus for defect analysis and process optimization based on artificial intelligence multi-modal fusion, characterized in that, The defect analysis and process optimization method based on artificial intelligence multi-modal fusion comprises: An acquisition module is configured to acquire a chip or device design layout, an expected process flow and parameters, and an expected electrical performance curve of a target semiconductor device in an inference stage; An extraction module is configured to input virtual data of the chip or device design layout, the expected process flow and parameters, and the expected electrical performance curve into an image encoder, a process encoder, and an electrical encoder respectively to extract corresponding feature vectors; A quantification module is configured to perform cross-modal information interaction on the corresponding feature vectors according to a cross-attention fusion mechanism to capture a corresponding relationship between a defect morphology, an electrical effect, and a process condition, generate a representation image from the corresponding relationship through a diffusion model, and quantify a responsibility degree of the process parameters according to the representation image to obtain fusion features; A determination module is configured to determine the representation image, a defect category, a root cause category, and a process parameter correction suggestion based on the fusion features; An optimization module is configured to issue the representation image, the defect category, the root cause category, and the process parameter correction suggestion through a manufacturing execution system (MES), an electronic design automation (EDA) tool, or a device interface to correct the process parameters when an analysis result meets a preset confidence condition, to obtain a closed-loop control result from defect analysis to process optimization.

7. The apparatus of claim 6, wherein, Further comprising: A data acquisition module is configured to acquire representation images, electrical curves, and process parameters of a target semiconductor device in a training stage before acquiring a chip or device design layout, an expected process flow and parameters, and an expected electrical performance curve of the target semiconductor device in an inference stage; A preprocessing module is configured to preprocess the representation images, the electrical curves, and the process parameters in the training stage respectively to obtain preprocessed data; A feature extraction module is configured to input the preprocessed data into the image encoder, the electrical encoder, and the process encoder respectively to extract the corresponding feature vectors; The multi-modal fusion and causal inference module models interactions of the corresponding feature vectors according to the cross-attention fusion mechanism to capture a corresponding relationship among the defect morphology, the electrical effect and the process condition, and simulates changes in model prediction results under preset conditions by using a counterfactual attribution mechanism to quantify a contribution degree of the process parameter to defect generation according to the changes; The physical consistency constraint module corrects the prediction results by using a preset electrical proxy model or a preset thermodynamic proxy model based on the corresponding relationship and the contribution degree to generate a constraint result satisfying a physical law of a target semiconductor device; A loss function is back-propagated to a preset model by a gradient based on the constraint result to jointly optimize a multi-modal encoder, a fusion layer and a prediction head to generate a process optimization result.

8. An electronic device, comprising: The method comprises: A memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the defect analysis and process optimization method based on artificial intelligence multi-modal fusion according to any one of claims 1-5.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the defect analysis and process optimization method based on artificial intelligence multi-modal fusion according to any one of claims 1-5.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed to implement the defect analysis and process optimization method based on artificial intelligence multi-modal fusion according to any one of claims 1-5.

Images