A wafer surface defect characterization method driven by multi-modal fusion domain migration features

Abstract

The application relates to the technical field of semiconductor manufacturing, and discloses a wafer surface defect characterization method driven by multi-modal fusion domain migration features, which comprises the following steps: firstly, multi-modal data is subjected to feature extraction and fusion, and then comprehensive feature alignment is carried out through a multi-level domain adversarial network. The aligned features are explicitly separated into domain-invariant and domain-variable parts; then, the domain-invariant features are enhanced through a self-attention mechanism, and minimum information loss reorganization is carried out with the aligned multi-modal features to construct a high-quality final feature set. Based on this, a multi-task learning framework can output the type, measurement and positioning results of defects in parallel. Through multi-level domain adversarial and fine feature decoupling and reorganization strategies, the domain offset problem caused by production batches or equipment replacement is effectively overcome, and accurate, robust and comprehensive intelligent characterization of wafer surface defects is realized.

Description

Technical Field

[0001] This invention relates to the field of semiconductor manufacturing technology, specifically to a method for characterizing wafer surface defects driven by multimodal fusion domain migration features. Background Technology

[0002] In semiconductor manufacturing, the surface quality of wafers directly determines the yield and performance of the final chip. Therefore, rapid and accurate detection and characterization of wafer surface defects are crucial for ensuring product quality. Currently, deep learning-based machine vision technology has become the mainstream method for achieving automated defect detection.

[0003] However, in actual industrial production, deep learning models often face a severe domain shift challenge. Due to subtle variations in production equipment, processing batches, material properties, or environmental parameters, wafer data collected at different times or on different production lines can exhibit significant differences in feature distribution. This causes a model trained in one data domain (source domain) to experience a sharp decline in performance when directly applied to another data domain (target domain), severely limiting the model's generalization ability and robustness.

[0004] To address this issue, existing technologies have attempted to introduce domain adaptation methods, but their strategies are often not deep enough. For example, some methods only perform general alignment at the final fused feature level, ignoring the domain offset problem that also exists within each individual modality of data before fusion. This "single-point" alignment cannot comprehensively eliminate the distribution differences between different data domains, resulting in limited generalization ability of the model when faced with unfamiliar product models or processing batches, making it difficult to meet complex production needs.

[0005] Furthermore, existing technologies also fall short in terms of the granularity of feature utilization. They typically lack an explicit mechanism to distinguish the intrinsic attributes of features, meaning they cannot clearly identify which features are common features that exist across all domains and are beneficial for defect characterization, and which features are merely personalized noise that can distinguish the data source. This ambiguity makes it difficult for models to construct a high-quality feature set, affecting the final accuracy and robustness in multi-variety, multi-category defect detection tasks.

[0006] Furthermore, from an application efficiency perspective, a complete characterization of a defect in industry typically requires information on multiple dimensions, such as its type, size, and spatial location. Current practices often involve separating these tasks and using multiple independent models or systems to handle them separately. This approach is not only cumbersome and inefficient, but also fails to provide a unified and comprehensive view of defect analysis. Summary of the Invention

[0007] The technical problem that this invention aims to solve is that existing surface defect detection methods, when dealing with multiple types of products and defects, mainly rely on extracting domain-invariant features, while not adequately considering the fusion of domain-variant features. This results in insufficient detection accuracy and robustness in scenarios with large inter-domain differences and few common features.

[0008] To address the aforementioned technical problems, the first aspect of this invention provides a method for characterizing wafer surface defects driven by multimodal fusion domain migration features, the method comprising the following steps:

[0009] S1. Acquire multimodal data of wafer surface defects, process the multimodal data using a multimodal feature extractor to extract single-modal features corresponding to each modality; subsequently, fuse the single-modal features to generate fused multimodal features. In one embodiment, the specific process of fusing the single-modal features is as follows: apply a multi-head self-attention mechanism, which takes different single-modal features from the same source domain as input, and dynamically integrates complementary information between different modalities by calculating the dot product attention of query, key, and value vectors, thereby outputting the fused multimodal features.

[0010] S2. A multi-level domain adversarial device is used to align the single-modal features and the fused multimodal features generated in step S1 to obtain aligned single-modal features and aligned fused multimodal features.

[0011] Based on these aligned features, quantization separation is performed to obtain domain-invariant features and domain-variant features. In one embodiment, the specific process of feature alignment is as follows:

[0012] Independent domain classifiers are configured for the unimodal features and the fused multimodal features, respectively, and a gradient inversion layer connects the multimodal feature extractor and the domain classifiers. During model training, this gradient inversion layer inverts the gradient sign during backpropagation, thereby maximizing the domain classification loss while minimizing the main task loss of the multimodal feature extractor, ultimately generating domain-confusing features. Specifically, for the adversarial training of the fused multimodal features, the adversarial loss... Calculated using the following formula:

[0013] ;

[0014] in, For the input sample, The total number of domains. For the true domain labels of the samples, For indicator functions, For domain classifiers that integrate multimodal features, For the sample Multimodal features after fusion.

[0015] In one embodiment, the specific process of obtaining the domain-invariant features through quantization separation is as follows:

[0016] Perform a Fourier transform on the spatiotemporal domain data portion of the aligned single-modal features, and extract the phase value information of the Fourier transform result. Use this phase value information as a domain-invariant feature characterizing the cross-domain common structure.

[0017] In one embodiment, the specific process of obtaining the domain-variable features through quantization separation is as follows:

[0018] A multidimensional Gaussian distribution hypothesis is established for the data distribution of the aligned fused multimodal features. By calculating and comparing the differences between the mean vector and covariance matrix of the feature distributions in different domains, domain-variable features that characterize information in a specific domain are extracted.

[0019] S3. Execute a defect domain migration feature reorganization strategy based on minimum information loss. This strategy enhances the domain-invariant features separated in step S2 and suppresses the domain-variant features, and finally reorganizes them to obtain a set of domain migration information features with minimum loss.

[0020] In one embodiment, the specific process of enhancing the domain-invariant feature is as follows:

[0021] A self-attention mechanism is applied to calculate the correlation between elements within the domain-invariant features, and based on the calculated correlation, the attention weight of common features that stably appear in all domains is increased.

[0022] In one embodiment, the specific process for suppressing the domain-variant feature is as follows:

[0023] By using a gradient inversion layer, the weights of personalized features that contribute little to the defect representation task but have high discriminative power for the domain origin are reduced during training.

[0024] S4. Based on the minimum loss domain migration information feature set obtained in step S3, it is processed through a multi-task learning fine-tuning strategy to obtain the defect type, defect measurement and defect location of the wafer surface defect.

[0025] In one embodiment, the specific process of fine-tuning the multi-task learning strategy is as follows:

[0026] The minimum loss domain transition information feature set is input into a shared feature encoder;

[0027] The encoded features output by the feature encoder are input in parallel to three independent task heads: a defect classification head, a defect measurement head, and a defect location head.

[0028] By jointly optimizing these three task heads, the model can simultaneously output three types of defect representation results. The task loss from this joint optimization... The calculation is performed using the following formula:

[0029] ;

[0030] in, The defect classification loss is calculated by the defect classification head. The defect measurement loss is calculated by the defect measurement head. The defect location loss is calculated by the defect location head. , , These are the preset weighting coefficients used to balance the three representation results.

[0031] This invention provides a wafer surface defect characterization method driven by multimodal fusion domain migration features. It has the following beneficial effects:

[0032] 1. This invention constructs a minimum-loss domain migration information feature set containing cross-domain common information and necessary personalized information by explicitly quantifying, separating, enhancing, suppressing, and recombining domain-invariant and domain-variant features. This feature set strengthens common features beneficial to defect characterization while weakening useless personalized features that can only distinguish domain origins, thereby improving the accuracy and robustness of defect characterization when handling multi-variety and multi-category defect detection tasks.

[0033] 2. This invention adopts a multi-level domain adversarial strategy, performing feature alignment at both the single-modal and multi-modal levels. This hierarchical alignment method can more comprehensively eliminate domain shifts caused by different product models or processing batches, effectively reducing the distribution differences between different data domains, thereby improving the model's generalization ability to unseen target domains.

[0034] 3. This invention introduces a multi-task learning fine-tuning strategy, using a single model to drive three parallel task heads: defect classification, measurement, and localization. This strategy enables the model to achieve comprehensive analysis of defect type, size, and spatial location based on a unified feature representation, providing a comprehensive and integrated defect representation scheme and improving detection efficiency. Attached Figure Description

[0035] Figure 1 This is a flowchart of a wafer surface defect characterization method driven by multimodal fusion domain migration features according to an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram of the system implementation environment according to an embodiment of the present invention;

[0037] Figure 3 This is a structural design diagram of a multimodal feature extractor in one embodiment of the present invention;

[0038] Figure 4 This is a structural design diagram of a multi-level domain adversary in one embodiment of the present invention;

[0039] Figure 5 This is a flowchart of a defect domain migration feature reconstruction algorithm with minimum information loss according to an embodiment of the present invention;

[0040] Figure 6 This is a schematic diagram of the feature enhancement module according to an embodiment of the present invention;

[0041] Figure 7 This is a schematic diagram of the feature recombination module according to an embodiment of the present invention;

[0042] Figure 8 This is a schematic diagram of the technical route of the surface defect detection method driven by multimodal fusion domain migration features of the present invention. Detailed Implementation

[0043] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] See attached document Figure 1 Appendix Figure 5 and attached Figure 8 , Figure 1 This is a flowchart of a wafer surface defect characterization method driven by multimodal fusion domain migration features according to an embodiment of the present invention. Figure 5 This is a flowchart of a defect domain relocation feature reconstruction algorithm with minimum information loss according to an embodiment of the present invention. Figure 8 This is a schematic diagram of the technical route of the multimodal fusion domain migration feature-driven surface defect detection method of the present invention. The present invention provides a wafer surface defect characterization method driven by multimodal fusion domain migration features, which may include the following steps:

[0045] S100. Based on the multimodal data of wafer surface defects, the single-modal features of each mode are extracted by a multimodal feature extractor, and the single-modal features are fused to obtain fused multimodal features.

[0046] S200. A multi-level domain adversarial device is used to align the single-modal features and the fused multimodal features to obtain aligned single-modal features and aligned fused multimodal features. Based on the aligned features, domain-invariant features and domain-variable features are obtained by quantization separation.

[0047] S300. Execute a defect domain migration feature reorganization strategy based on minimum information loss, enhance the domain-invariant features, and suppress the domain-variant features to reorganize and obtain a set of domain migration information features with minimum loss.

[0048] S400. Based on the minimum loss domain migration information feature set, the defect type, defect measurement, and defect location of the wafer surface defect are obtained through a multi-task learning fine-tuning strategy.

[0049] The method of this invention can be executed by an electronic device or a computing system. The device or system may be configured with one or more processors (e.g., a central processing unit (CPU) or a graphics processing unit (GPU), a memory, and a non-transitory computer-readable storage medium for storing computer program instructions. The processor executes the computer program instructions stored on the storage medium to implement the steps of S100 to S400 described above.

[0050] The method described in this embodiment of the invention is executed by an electronic device or a computing system. (See attached diagram.) Figure 2 , Figure 2 This is a schematic diagram of a system implementation environment according to an embodiment of the present invention. The electronic device or computing system may be configured with one or more processors 21, a memory 22, and one or more non-transitory computer-readable storage media 23 for storing computer program instructions.

[0051] The processor 21 can be a general-purpose central processing unit (CPU), or a graphics processing unit (GPU) or application-specific integrated circuit (ASIC) specifically configured to accelerate neural network calculations. The memory 22 serves as a temporary data storage area for the processor 21 during operation. The non-transitory computer-readable storage medium 23 permanently stores computer program instructions; when the processor 21 executes these instructions, the defect characterization method described in this embodiment of the invention is implemented.

[0052] In one specific embodiment, the training and deployment phases of the method can be executed in different system environments. For the training phase, the electronic device or computing system is one or more servers configured with high-performance graphics processing units (GPUs) to support model training and parameter optimization on large-scale multimodal data.

[0053] During the deployment phase, the method of this invention can be deployed on an edge computing device or an industrial control computer integrated with a wafer inspection device. In this scenario, the electronic device or computing system receives a real-time multimodal data stream from the inspection device, executes a pre-trained model for inference calculations, and outputs characterization results of wafer surface defects to achieve on-site, real-time defect analysis.

[0054] In a specific implementation scenario, the input to the method described in this embodiment of the invention, namely multimodal data, is acquired from the semiconductor wafer processing and inspection process. The multimodal data includes, but is not limited to, the following types:

[0055] Two-dimensional optical microscopic image data is used to characterize the planar morphology of defects on the wafer surface; three-dimensional morphology scanning data is used to characterize the depth and volume information of defects.

[0056] Equipment vibration timing signals are used to characterize the mechanical stability of the processing process;

[0057] Acoustic emission timing signals are used to characterize the acoustic energy released when a material undergoes micro-cracks or deformation.

[0058] Machining force timing signal, used to characterize the mechanical state of the interaction between the tool or grinding head and the wafer surface;

[0059] Temperature sensor data is used to characterize the thermal environment of the processing area.

[0060] In the technical solution of this invention embodiment, a "domain" is defined as a set of datasets with the same data distribution characteristics. Specifically, data sets collected from different product models, different production lines, different process batches, or different physical environments are all considered to be data from different domains.

[0061] Due to the differences between domains, directly using a model trained on one or more source domains to detect data in the target domain can lead to performance degradation due to inconsistent data distribution. The technical solution of this invention aims to improve the model's defect representation performance in unseen target domains by learning from multi-source domain data.

[0062] In step S100, the acquired raw multimodal data is first processed by a multimodal feature extractor. The multimodal feature extractor consists of multiple parallel feature extraction branches, and each modality of data is processed by an independent, specially configured feature extraction branch.

[0063] See attached document Figure 3The attached figure is a structural design diagram of a multimodal feature extractor in one embodiment of the present invention, designed to achieve scale alignment of features. For input data of different modalities such as images and time series, the corresponding feature extraction branches all adopt a deep network structure based on residual connections. Specifically, the input features first pass through a convolutional layer, followed by a batch normalization layer, and then through a nonlinear activation function layer (e.g., ReLU). This processing sequence constitutes a basic convolutional unit. A residual module consists of two cascaded basic convolutional units, and there is a skip connection between the input and output of the module. This connection directly adds the input of the module to the output of the module to facilitate gradient propagation and allow the network to learn deeper features. In one embodiment, the feature extraction branch may be composed of four such residual modules stacked together.

[0064] After the deep features are extracted, they pass through a channel enhancement layer. The function of this layer is to recalibrate the weights of different channels of the feature map to enhance the expression of key feature channels.

[0065] In step S100, after the multimodal feature extractor generates the single-modal features corresponding to each modality, the method of this embodiment of the invention then fuses the group of single-modal features. (Refer to the appendix...) Figure 3 The attached figure is a structural design diagram of a multimodal feature extractor in one embodiment of the present invention.

[0066] This fusion process is achieved through a multi-head self-attention mechanism. For different modalities (e.g., visual features) from the same source domain, after being processed by their respective feature extraction branches... Harmony emission characteristics This module first projects each unimodal feature through its own independent learnable linear transformation layers to generate a query vector, key vector, and value vector specific to that modality (e.g., for visual features, generating...). ).

[0067] Subsequently, the query vectors, key vectors, and value vectors generated from all modalities are concatenated to form a unified query matrix. A fused key matrix A fused value matrix :

[0068] Concat ;

[0069] Concat ;

[0070] Concat ;

[0071] Finally, the fused , , The input is fed into a dot-product attention computation unit to calculate the fused feature. This feature can be further processed through one or more fully connected layers to generate the final fused multimodal feature that incorporates cross-modal complementary information. At this point, step S100 has been completed.

[0072] In step S200, the method first performs feature alignment processing on the single-modal features and the fused multimodal features generated in step S100. In the technical solution of this invention, to improve the model's generalization ability to unseen target domains, a training framework based on meta-learning is adopted. The source domain data is divided into meta-train domains and meta-test domains.

[0073] See attached document Figure 4 The attached figure is a structural design diagram of a multi-level adversary in one embodiment of the present invention. This adversary aims to utilize the complementary specific features between different domains to perform multi-level domain adversarial operations on the same mode in different source domains.

[0074] Specifically, the adversarial mechanism establishes three parallel adversarial paths:

[0075] Single-modal adversarial path (visual): Visual features from the meta-training domain and visual features from the meta-test domain are concatenated and input into a gradient reversal layer and a subsequent domain classifier to compute the domain adversarial loss for the visual modality. .

[0076] Single-modal adversarial path (acoustic emission): Acoustic emission features from the meta-training domain and the meta-test domain are concatenated and input into the gradient inversion layer and the domain classifier to calculate the domain adversarial loss of the acoustic emission modality. .

[0077] The multimodal adversarial path is fused as follows: First, the visual and acoustic emission features from the meta-training domain are fused using a multi-head self-attention module; similarly, the two features from the meta-test domain are also fused. Then, the two fused features are concatenated and input into a gradient reversal layer and a domain classifier to calculate the domain adversarial loss for multimodal fusion. .

[0078] In one specific embodiment, the loss function adopts the standard classification cross-entropy loss, and its specific calculation formula is as follows:

[0079] ;

[0080] in, For the input sample, The total number of domains. For the true domain labels of the samples, This is an indicator function (1 if the condition is true, 0 otherwise). For domain classifiers that integrate multimodal features, For the sample Multimodal features after fusion

[0081] Through the gradient inversion layer, the feature extractor is optimized during training to generate features that can confuse the three domain classifiers, thereby forcing the model to learn domain-independent feature representations that are generalizable at both the unimodal and fusion modality levels.

[0082] In step S200, after the feature alignment process is completed, the method of this embodiment of the invention then processes the aligned single-modal features to quantize and separate the domain-invariant features contained therein. This process is mainly for single-modal features with spatiotemporal domain structures, such as features derived from image data or time-series signals.

[0083] In one specific embodiment, the quantization process is implemented by performing a Fourier transform on the aligned single-modal feature data. For a given single-modal feature vector or feature map with a spatiotemporal structure... Perform Fourier transform on it To obtain its representation in the frequency domain This process can be represented by the following formula:

[0084] ;

[0085] in, Representing time-domain or space-domain variables, Represents frequency domain variables, yes The Fourier transform result.

[0086] The result of the Fourier transform It is a complex function that can be decomposed into its amplitude spectrum. and its phase spectrum In one embodiment of the present invention, the core of the method lies in extracting and utilizing its phase spectrum information while ignoring or discarding its amplitude spectrum information.

[0087] The technical basis of this operation is that the structural information of features (such as the shape, contour, or specific patterns in timing signals of wafer surface defects) is mainly encoded in the phase spectrum. In the middle, and the amplitude spectrum This reflects more the energy or intensity information of the features (such as the brightness and contrast of an image or the amplitude of a signal), which often varies significantly between different data domains. Therefore, by retaining only the phase spectrum information, it is possible to effectively separate those common features that remain stable and consistent across different domains and are related to specific defect structures.

[0088] Finally, the extracted Fourier phase information, or the signal reconstructed based on this phase information, is quantized into the domain-invariant features to be obtained in this step (which will be further refined in subsequent steps). (This is indicated by the text). The domain-invariant feature will then be used for feature reorganization in step S300.

[0089] In step S200, in parallel with the quantization of domain-invariant features, the method of this embodiment of the invention also processes the aligned fused multimodal features to quantize and separate the domain-variant features contained therein. Domain-variant features refer to information that can characterize the unique properties of a specific data domain.

[0090] In one specific embodiment, the quantization process is based on the assumption of a multidimensional normal distribution (also known as a multidimensional Gaussian distribution) for the characteristic distribution of each domain. Specifically, for each domain... Collect all aligned and fused multimodal feature vectors belonging to this domain to form a set. Subsequently, for this set Fit a multidimensional Gaussian distribution and calculate its statistical parameters, i.e., the mean vector. Covariance Matrix .

[0091] The mean vector The calculation formula is as follows:

[0092] ;

[0093] in, It is a set The number of feature vectors in the middle. It is a fused multimodal feature vector in the set. The mean vector Represents the domain The central location of the characteristic distribution.

[0094] The covariance matrix The calculation formula is as follows:

[0095] ;

[0096] in, This represents the transpose operation of a vector. The covariance matrix... Describes the domain The degree of dispersion of the feature distribution and the correlation between features in each dimension, i.e. the shape and direction of the distribution.

[0097] Through the above calculations, the differences in characteristic distributions between different domains are quantified as differences in their respective Gaussian distribution parameters (mean vector and covariance matrix). Therefore, this set of statistical parameters calculated from all domains... This is considered as the quantified domain-variable feature. It should be noted that this quantification result is mainly used for explicit analysis and measurement of inter-domain differences, while the suppression function of domain-variable features is implicitly achieved by the aforementioned multi-level domain adversarial training mechanism. At this point, step S200 is complete.

[0098] In step S300, the method of this embodiment of the invention executes a defect domain migration feature reorganization strategy based on minimum information loss to generate a final minimum-loss domain migration information feature set for defect characterization. This strategy mainly includes enhancing domain-invariant features and reorganizing the enhanced domain-invariant features with aligned multimodal features.

[0099] The method first performs enhancement processing on the domain-invariant features obtained in step S200. The purpose of this processing is to further improve the saliency of those feature elements that are most critical to defect characterization while preserving cross-domain commonalities.

[0100] See attached document Figure 6 , Figure 6 This is a schematic diagram of the structure of a feature enhancement module according to an embodiment of the present invention. In a specific embodiment, the enhancement process is implemented through a self-attention module. The self-attention module receives the domain-invariant features quantized from Fourier phase information. As input.

[0101] The self-attention module first processes the input features through three independent learnable linear transformation layers. Perform linear projections separately to generate the query matrix. Key matrix Sum matrix This process can be represented by the following formula:

[0102] ;

[0103] ;

[0104] ;

[0105] in, , , These are the learnable weight matrices used to generate queries, keys, and values.

[0106] The generated matrix is ​​then input into a dot product attention computation unit, which calculates the correlation scores between elements within the features to evaluate the value matrix. We perform a weighted summation to obtain the enhanced field-invariant features. The calculation process is represented by the following formula:

[0107] ;

[0108] in, It is a key matrix The transpose of the matrix, It is the dimension of the vectors in the key matrix, used as a scaling factor. It is a normalization function.

[0109] By performing this self-attention operation, the domain-invariant features Each feature element within the process is recalculated based on its correlation with all other elements. The output of this process is an enhanced domain-invariant feature. In this feature, the weights of feature elements that are stable in all domains and contribute significantly to representing the essential structure of defects are increased, while the weights of redundant or secondary information within the feature are decreased.

[0110] In step S300, corresponding to the enhancement processing of domain-invariant features, the method of this embodiment of the invention further includes a suppression processing of the domain-variant features. This processing aims to weaken feature components that are only related to the source of the data domain but are not beneficial to the final defect characterization task.

[0111] This suppression process is not a module executed independently in step S300, but rather a functional effect achieved through the multi-level domain adversarial training mechanism described in step S200. Specifically, the gradient inversion layer, which connects the multimodal feature extractor and the various domain classifiers in the multi-level domain adversarial processor, performs a crucial function during the backpropagation process of model training.

[0112] As mentioned earlier, the gradient inversion layer multiplies the gradient value from the domain classifier by a negative constant before passing it to the front-end feature extractor. This operation causes the feature extractor to update its parameters in a direction that maximizes the loss function value of the domain classifier when optimizing its parameters.

[0113] The result of this process is that the feature extractor is constrained to avoid generating features that could easily distinguish the source of the data domain. In other words, any feature component that is highly correlated only with the characteristics of a specific domain (e.g., processing marks on a specific production line, material background of a specific batch) and unrelated to the characterization of the defect itself (e.g., defect type, size) is effectively weakened in the feature vector output by the feature extractor. Through this adversarial constraint, the task-irrelevant parts of the domain-variable features are suppressed; this process is also known as feature pruning.

[0114] In the final stage of step S300, the method of this embodiment performs feature recombination to generate a final feature set for subsequent processing. This recombination process integrates the enhanced domain-invariant features obtained in the preceding steps and the multimodal features after alignment and suppression processing.

[0115] See attached document Figure 7 , Figure 7 This is a schematic diagram of the feature recombination module according to an embodiment of the present invention. The purpose of this recombination operation is to combine domain-invariant information that represents the essential structure of defects and has cross-domain stability with multimodal information that contains rich context and has undergone domain alignment processing, so as to form a feature representation with comprehensive information.

[0116] The input to the recombination module consists of two parts: one is the enhanced domain-invariant feature. Secondly, aligned fused multimodal features. This feature is the output of the multimodal fusion module in step S100, and has been aligned through the domain adversarial training process described in step S200.

[0117] In one specific embodiment, the recombination operation is implemented through a feature splicing unit. This unit receives... and Two feature vectors are concatenated along a specific dimension to generate a composite feature vector with higher dimensionality and richer information. This process can be represented by the following formula:

[0118] Concat ;

[0119] Among them, Concat This represents a vector concatenation operation.

[0120] In another embodiment, in order to further integrate and reduce the dimensionality of the concatenated features, the composite feature vector... It can be input into one or more fully connected layers. The fully connected layers perform linear transformations and non-linear activations on the composite feature vector, and finally output a feature vector with higher integration density after dimensional adjustment.

[0121] The feature set output in this step This is the minimum loss domain-transfer information feature set. This strategy is called minimum information loss based because the domain-invariant structured information embedded in the Fourier phase spectrum, which is easily lost or weakened in standard feature extraction processes, is explicitly separated, enhanced, and ultimately reintegrated into the main feature stream by the method of this invention. This operation ensures that the final feature set used for defect characterization possesses both cross-domain generalization ability and detailed information specific to each sample. At this point, step S300 is complete.

[0122] In step S400, to obtain the final defect representation model, this embodiment of the invention employs a meta-learning optimization strategy based on minimum information loss. (See attached diagram.) Figure 7 This strategy aims to learn the initial parameters of a model with rapid adaptability by simulating the training process of domain migration.

[0123] In one specific embodiment, to simultaneously achieve the three representation objectives, this embodiment of the invention employs a multi-task defect representation module. This module receives the minimum loss domain transition information feature set generated in step S300 as input, and includes a shared feature backbone and three parallel task-specific decoding heads:

[0124] Classification Header: One or more fully connected layers that ultimately output the probability distribution of each type of defect through the Softmax function to determine the defect type.

[0125] Defect Measurement Head: A regression network (e.g., one or more fully connected layers) that outputs one or more continuous values ​​corresponding to the physical size of the defect.

[0126] Defect localization head: A segmentation network decoder (e.g., an upsampling network based on transposed convolution) outputs a mask map with the same size as the original image, achieving pixel-level localization.

[0127] Multi-task classification loss during model training It consists of the weighted sum of the losses from these three task heads:

[0128] ,in The defect classification loss is calculated by the defect classification head. The defect measurement loss is calculated by the defect measurement head. The defect location loss is calculated by the defect location head. , These are the preset weighting coefficients used to balance the three representation results.

[0129] Throughout the training process, the total loss function consists of classification loss and domain adversarial loss. Domain adversarial total loss. The weighted sum of the losses of the three adversarial paths defined in step S200 is obtained as follows:

[0130] ;

[0131] in, , , These are hyperparameters used to adjust the importance of visual, acoustic emission, and fusion modal adversarial loss, respectively.

[0132] The model optimization process follows a two-stage meta-learning update rule:

[0133] InnerLoopUpdate: In one iteration, the meta-training classification loss of the model is first calculated on the data in the meta-training domain. (This loss can be caused by the multi-task loss described in step S400, such as...) (Composition). In one specific embodiment, the total task loss The calculation is performed using the following formula:

[0134] ;

[0135] in, The defect classification loss is calculated by the defect classification head. The defect measurement loss is calculated by the defect measurement head. The defect location loss is calculated by the defect location head. , , These are the preset weighting coefficients used to balance the three representation results;

[0136] Based on this loss, the current model parameters Perform a simulated gradient descent update to obtain a temporary, updated set of meta-trained classification parameters. . ,in It is the learning rate of the internal loop.

[0137] OuterLoopUpdate: Next, use this set of temporary parameters. Evaluate the model's performance on the meta-test domain data and calculate the meta-test classification loss. Ultimately, this is used to update the original model parameters. Total objective loss function It consists of the meta-training classification loss, the meta-test classification loss, and the total domain adversarial loss: ,in , This is a hyperparameter used to balance the meta-test loss and the domain adversarial loss. The total loss function is minimized using optimization algorithms such as gradient descent. To update and optimize the original parameters .

[0138] This process drives the model to learn a more generalizable initial state by minimizing the loss on the meta-test domain. This enables the model to adapt quickly with very few fine-tunings when faced with a completely new and unseen target domain, thereby achieving defect domain feature reorganization and accurate and robust representation of defects based on minimum information loss.

[0139] In another implementation scenario of step S400, namely the model deployment and inference stage, the method of this embodiment of the invention deploys the final model, which has been trained offline and whose model parameters have been solidified, in the actual production environment.

[0140] In one specific embodiment, the deployment environment is an edge computing device or industrial control computer tightly integrated with wafer inspection equipment (e.g., scanning electron microscope, optical inspection instrument, atomic force microscope, etc.). The deployed system receives real-time multimodal data streams from the inspection equipment, which originate from a target domain that was not encountered during the training phase.

[0141] When a wafer sample data to be inspected is input, the deployed system performs a forward inference calculation. During this process, the system strictly follows the calculation path defined in steps S100 to S400, specifically including:

[0142] A multimodal feature extractor with fixed parameters is used to generate single-modal features for various types of input data.

[0143] The multimodal fusion module integrates single-modal features into a unified fused multimodal feature.

[0144] Perform Fourier transform on the single-mode features to extract their phase spectrum as domain-invariant features, and perform self-attention enhancement on the domain-invariant features;

[0145] The enhanced domain-invariant features are recombined with the fused multimodal features to generate the final minimum loss domain migration information feature set.

[0146] The feature set is input into the defect characterization module, which performs calculations via a shared backbone and parallel multi-task heads, and finally outputs three results simultaneously: a category representing the type of defect, a measurement value representing the size of the defect, and a localization mask map representing the location of the defect.

[0147] It is worth noting that during the inference phase, all modules used solely for training, such as the multi-level domain adversarial processor and its gradient inversion layer, are inactive and do not participate in computation. The entire inference process is a pure forward propagation, which is computationally efficient and can meet the real-time analysis requirements of industrial settings.

[0148] Because the method in this embodiment of the invention has a unique feature separation and recombination mechanism during the training phase, enabling the model to have high robustness to domain changes, the deployed system does not require any retraining or parameter fine-tuning for new target domain data to achieve high-precision defect characterization. This greatly reduces the cost of migrating the model between different production lines or processes, and improves the practicality and deployment efficiency of the intelligent defect detection solution. Step S400 is now complete.

[0149] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A wafer surface defect characterization method based on multi-modal fusion domain migration feature driving, characterized in that, The method comprises the following steps: S1, based on the multi-modal data of wafer surface defects, extracting single-modal features of each mode through a multi-modal feature extractor, and fusing the single-modal features to obtain fused multi-modal features; S2, using a multi-level domain adversarial network to align the single-modal features and the fused multi-modal features, obtaining aligned single-modal features and aligned fused multi-modal features, and quantitatively separating domain-invariant features and domain-variable features based on the aligned features; S3, performing a defect domain migration feature reorganization strategy based on minimum information loss, enhancing the domain-invariant features and suppressing the domain-variable features to reorganize a minimum loss domain migration information feature set; In step S3, the domain-invariant features are enhanced, specifically including: Using a self-attention mechanism to calculate the correlation between elements in the domain-invariant features, and accordingly increasing the attention weight of common features that stably appear in all domains; S4, based on the minimum loss domain migration information feature set, obtaining the defect type, defect measurement and defect positioning of the wafer surface defects through a multi-task learning fine-tuning strategy.

2. The multi-modal fusion domain migration feature driven wafer surface defect characterization method of claim 1, wherein, In step S1, the single-modal features are fused to obtain fused multi-modal features, specifically including: Using a multi-head self-attention mechanism, different modal features of the same source domain are inputted to dynamically calculate and fuse the complementary information between different modalities. 3.The wafer surface defect characterization method of claim 1, wherein, In step S2, the multi-level domain adversarial network is used for feature alignment, specifically including: For the single-modal features and the fused multi-modal features, independent domain classifiers are set up and gradient reversal layers are introduced for adversarial training, so that the multi-modal feature extractor generates domain-confused features.

4. The multi-modal fusion domain migration feature driven wafer surface defect characterization method of claim 3, wherein, In step S2, the domain-invariant features are obtained by quantitative separation, specifically including: Performing Fourier transform on the spatio-temporal domain data in the aligned single-modal features, and extracting the Fourier phase value information as the basis for quantitative mining of the domain-invariant features.

5. The multi-modal fusion domain migration feature driven wafer surface defect characterization method of claim 1, wherein, In step S2, the domain-variable features are obtained by quantitative separation, specifically including: Performing multi-dimensional Gaussian distribution assumption on the aligned fused multi-modal features, extracting the domain-variable features by capturing and comparing the mean and covariance differences of feature distribution between different domains.

6. The multi-modal fusion domain migration feature driven wafer surface defect characterization method of claim 1, wherein, In step S3, the domain-variable features are suppressed, specifically including: Through the gradient reversal layer, the weight of the personalized features that can distinguish the domain source but are not beneficial to the defect representation task is weakened in the training process.

7. The multi-modal fusion domain migration feature driven wafer surface defect characterization method of claim 1, wherein, In step S4, the multi-task learning fine-tuning strategy is used, specifically including: Inputting the minimum loss domain migration information feature set into a shared feature encoder, and parallelly inputting the encoded features into a defect classification head, a defect measurement head and a defect positioning head, and outputting three representation results at the same time through joint optimization.

8. The multi-modal fusion domain migration feature driven wafer surface defect characterization method of claim 7, wherein, the jointly optimized task loss is calculated by the following equation: ； wherein, is a defect classification loss calculated by the defect classification head, is a defect measurement loss calculated by the defect measurement head, is a defect localization loss calculated by the defect localization head, , , is a preset weight coefficient for balancing the three representation results.

9. The multi-modal fusion domain migration feature driven wafer surface defect characterization method of claim 3, wherein, The adversarial training for the fusion multi-modal feature, an adversarial loss is calculated by the following formula: ； wherein, is the input sample, is the total number of domains, is the true domain label of the sample, is the indicator function, is the domain classifier for fused multi-modal features, is the sample fused multi-modal features.

Images