Training method of retrieval model, retrieval method and electronic device

Abstract

Embodiments of the present application disclose a training method of a retrieval model, a retrieval method and an electronic device. The training method comprises: fusing prior knowledge of historical wafer defect cases in a database into a multi-modal feature vector of a current training sample through a cross-attention module to generate a multi-modal enhanced query vector of the current training sample; splicing enhanced query vectors of different modes of the current training sample into a multi-modal sequence of the current training sample according to a time sequence of a process flow through a vector splicing module; inputting the multi-modal sequence of the current training sample into a sequence processing module to extract fusion features of different modes of the current training sample; calculating a distribution distance of the fusion features of different modes of the current training sample in a same feature space to construct a global alignment loss function, and optimizing parameters of the retrieval model through a back propagation algorithm. Embodiments of the present application improve the retrieval accuracy of historical wafer defect cases.

Description

Technical Field

[0001] This application relates to the field of semiconductor manufacturing technology, and in particular to a method for training a retrieval model, a retrieval method, and an electronic device. Background Technology

[0002] In semiconductor wafer manufacturing, defect detection is a crucial step in ensuring product yield. As manufacturing processes continue to shrink to the nanometer scale, the types and details of wafer surface defects are becoming increasingly complex and subtle, placing higher demands on the accuracy and efficiency of defect analysis. Currently, the industry generally uses defect images and defect maps generated by automated inspection equipment as the primary basis for analysis, combined with multi-source information such as process parameters and equipment logs for root cause determination. Traditional methods mainly rely on deep learning-based image recognition models (such as CNN and YOLO) or engineer experience for defect classification and tracing. Historical cases are often stored in offline document formats such as PPTs and Excel, resulting in a fragmented and unstructured knowledge management approach.

[0003] Current wafer defect analysis technologies heavily rely on large amounts of labeled data, but clearly marked defect samples are extremely scarce in actual production, especially for rare defect types. Due to their low frequency of occurrence, it is difficult to accumulate sufficient training samples. Furthermore, the accumulated knowledge of cases where defect root causes lead to yield declines is mostly stored in engineers' minds or in unstructured documents, resulting in fragmented knowledge, inconsistent formats, and difficulties in structured storage and retrieval. This also hinders cross-case correlation analysis and knowledge consolidation. The high cost of training engineers and the risk of employee turnover further exacerbate the knowledge loss problem, making the defect analysis process highly subjective and inefficient. Summary of the Invention

[0004] In view of the above problems, the purpose of this application is to provide a training method, retrieval method and electronic device for a retrieval model, which aims to solve the problems of defect analysis in wafer manufacturing scenarios that rely on human experience, insufficient multimodal information fusion, weak generalization ability with small samples and difficulty in knowledge accumulation and reuse, and improve the retrieval accuracy of historical wafer defect cases.

[0005] According to a first aspect of the embodiments of this application, a training method for a retrieval model for wafer defect cases is provided, the retrieval model including a cross-attention module, a vector concatenation module, and a sequence processing module, the training method including:

[0006] The cross-attention module integrates prior knowledge of historical wafer defect cases in the database into the multimodal feature vector of the current training sample to generate the multimodal enhanced query vector of the current training sample.

[0007] The vector concatenation module concatenates the enhanced query vectors of different modalities of the current training sample into a multimodal sequence of the current training sample according to the time sequence of the process flow.

[0008] The multimodal sequence of the current training sample is input into the sequence processing module to extract the fusion features of different modalities of the current training sample;

[0009] The distribution distance of the fusion features of different modalities of the current training sample in the same feature space is calculated to construct a global alignment loss function, and the parameters of the retrieval model are optimized by the backpropagation algorithm.

[0010] Optionally, before incorporating prior knowledge of historical wafer defect cases from the database into the multimodal feature vector of the current training sample through the cross-attention module to generate the multimodal enhanced query vector of the current training sample, the training method further includes:

[0011] The database is constructed, which includes multiple data units. Each data unit is formed by encoding multimodal raw data of historical wafer defect cases, and each data unit includes a key vector and a value vector.

[0012] Optionally, in the database, for each historical wafer defect case, its multimodal feature vector is extracted, and these feature vectors are aligned according to the corresponding process time steps to form a time-series data unit.

[0013] Optionally, the step of integrating prior knowledge of historical wafer defect cases in the database into the multimodal feature vector of the current training sample through the cross-attention module to generate the multimodal enhanced query vector of the current training sample includes:

[0014] The multimodal raw data of the current training sample are input into the corresponding pre-trained encoder to extract the multimodal feature vector of the current training sample;

[0015] Extract the set of key vectors and the set of value vectors for historical wafer defect cases from the database;

[0016] The attention weight between the multimodal feature vector of the current training sample and the key vector set is calculated using a cross-attention mechanism.

[0017] The value vector set is weighted and aggregated based on the attention weights, and the prior knowledge of the most relevant historical wafer defect cases is incorporated into the multimodal feature vector of the current training sample to generate the multimodal enhanced query vector of the current training sample.

[0018] Optionally, the step of concatenating the enhanced query vectors of different modalities of the current training sample into a multimodal sequence of the current training sample according to the time sequence of the process flow through the vector concatenation module includes:

[0019] The enhanced query vectors of different modalities of the current training sample are divided into time steps to obtain the process stages corresponding to the enhanced query vectors of different modalities of the current training sample.

[0020] Based on the time sequence of the process flow, the modal augmented query vectors at the same time step are alternately concatenated to form the multimodal sequence of the current training sample.

[0021] Optionally, the enhanced query vectors of different modalities of the current training sample include image modality enhanced query vectors and text modality enhanced query vectors. The step of alternately concatenating the enhanced query vectors of each modality at the same time step according to the time sequence of the process flow to form the multimodal sequence of the current training sample includes:

[0022] The image modality augmented query vector and the text modality augmented query vector at the i-th time step are alternately concatenated to generate the concatenated augmented query vector at the i-th time step, where i is an integer greater than 0.

[0023] Based on the time sequence of the process flow, the concatenation enhancement query vectors of multiple time steps in the process flow are concatenated sequentially to generate the multimodal sequence of the current training sample.

[0024] Optionally, the step of inputting the multimodal sequence of the current training sample into the sequence processing module and extracting the fusion features of different modalities of the current training sample includes:

[0025] The multimodal sequence is input into the sequence processing module based on the state-space mechanism, and the sequence processing module performs time-series modeling on the multimodal sequence through the state-space model architecture;

[0026] During forward propagation, the sequence processing module introduces an optimal transmission strategy to perform fine-grained alignment of the hidden states of different modalities.

[0027] The sequence processing module outputs the fusion features of different modalities of the current training sample after cross-modal alignment optimization.

[0028] Optionally, the fusion features of different modalities of the current training sample include semantic enhancement features of the defect image modality, temporal fusion features of the equipment process parameter modality, and context-aware features of the defect description text modality. The step of calculating the distribution distance of the fusion features of different modalities of the current training sample in the same feature space to construct a global alignment loss function, and optimizing the parameters of the wafer defect case retrieval model through a backpropagation algorithm, includes:

[0029] Calculate the first maximum mean difference between the semantic enhancement features of the defect image modality and the context-aware features of the defect description text modality of the current training sample in the feature space;

[0030] Calculate the second maximum mean difference between the temporal fusion features of the equipment process parameter modality and the context-aware features of the defect description text modality of the current training sample in the feature space;

[0031] The first maximum mean difference and the second maximum mean difference are weighted and summed to form the global alignment loss function;

[0032] The global alignment loss function is minimized using the backpropagation algorithm, and the parameters of the cross-attention module, the vector concatenation module, and the sequence processing module are optimized until the model converges.

[0033] According to a second aspect of the embodiments of this application, a method for retrieving wafer defect cases is provided, applied to a retrieval model trained according to the above-described training method, the retrieval method comprising:

[0034] The multimodal feature vector of the wafer defect case to be retrieved is input into the trained retrieval model. Through the cross attention module, the prior knowledge of historical wafer defect cases in the database is integrated into the multimodal feature vector of the wafer defect case to be retrieved, thereby generating the multimodal enhanced query vector of the wafer defect case to be retrieved.

[0035] The vector splicing module splices the enhanced query vectors of different modes of the wafer defect cases to be retrieved into a multimodal sequence of the wafer defect cases to be retrieved according to the time sequence of the process flow.

[0036] The multimodal sequence of the wafer defect case to be retrieved is input into the sequence processing module to extract the fusion features of different modes of the wafer defect case to be retrieved;

[0037] The fusion features of different modes of the wafer defect case to be retrieved are used to perform similarity matching and sorting with the multimodal fusion features of historical wafer defect cases in the database, and one or more historical wafer defect cases with the highest similarity are returned as the retrieval results.

[0038] According to a third aspect of the embodiments of this application, a training apparatus for a retrieval model for wafer defect cases is provided, the retrieval model including a cross-attention module, a vector concatenation module, and a sequence processing module, the training apparatus including:

[0039] The first vector enhancement unit is used to integrate prior knowledge of historical wafer defect cases in the database into the multimodal feature vector of the current training sample through the cross-attention module, and generate the multimodal enhanced query vector of the current training sample.

[0040] The first vector splicing unit is used to splice the enhanced query vectors of different modalities of the current training sample into a multimodal sequence of the current training sample according to the time sequence of the process flow through the vector splicing module.

[0041] The first vector fusion unit is used to input the multimodal sequence of the current training sample into the sequence processing module and extract the fusion features of different modalities of the current training sample;

[0042] The parameter optimization unit is used to calculate the distribution distance of the fusion features of different modalities of the current training sample in the same feature space, thereby constructing a global alignment loss function and optimizing the parameters of the retrieval model through the backpropagation algorithm.

[0043] According to a fourth aspect of the embodiments of this application, a retrieval device for wafer defect cases is provided, applied to a retrieval model trained according to the training method described above, the retrieval device comprising:

[0044] The second vector enhancement unit is used to input the multimodal feature vector of the wafer defect case to be retrieved into the trained retrieval model. Through the cross attention module, the prior knowledge of historical wafer defect cases in the database is integrated into the multimodal feature vector of the wafer defect case to be retrieved, thereby generating the multimodal enhanced query vector of the wafer defect case to be retrieved.

[0045] The second vector splicing unit is used to splice the enhanced query vectors of different modes of the wafer defect case to be retrieved into a multimodal sequence of the wafer defect case to be retrieved according to the time sequence of the process flow through the vector splicing module.

[0046] The second vector fusion unit is used to input the multimodal sequence of the wafer defect case to be retrieved into the sequence processing module and extract the fusion features of different modes of the wafer defect case to be retrieved.

[0047] The similarity matching unit is used to perform similarity matching and sorting by using the fusion features of different modes of the wafer defect case to be retrieved and the multimodal fusion features of historical wafer defect cases in the database, and return one or more historical wafer defect cases with the highest similarity as the retrieval results.

[0048] According to a fifth aspect of the embodiments of this application, an electronic device is provided, comprising:

[0049] One or more processors;

[0050] A memory for storing executable instructions that, when executed by the one or more processors, cause the electronic device to perform the methods described above.

[0051] The unexpected technical effect of this application is as follows: By using a cross-attention module, prior knowledge of historical wafer defect cases in the database is integrated into the multimodal feature vector of the current training sample to generate a multimodal enhanced query vector for the current training sample. By using a vector concatenation module, the enhanced query vectors of different modalities of the current training sample are concatenated into a multimodal sequence of the current training sample according to the time sequence of the process flow. The multimodal sequence of the current training sample is input into a sequence processing module to extract the fusion features of different modalities of the current training sample. The distribution distance of the fusion features of different modalities of the current training sample in the same feature space is calculated, thereby constructing a global alignment loss function. The parameters of the retrieval model are optimized through a backpropagation algorithm. In this way, using the trained retrieval model to retrieve historical wafer defect cases solves the problems of defect analysis relying on human experience, insufficient multimodal information fusion, weak generalization ability with small samples, and difficulty in knowledge accumulation and reuse in wafer manufacturing scenarios, thereby improving the retrieval accuracy of historical wafer defect cases.

[0052] Furthermore, a database is constructed, comprising multiple data units. Each data unit is formed by encoding multimodal raw data of historical wafer defect cases. Each data unit includes key vectors and value vectors. The multimodal raw data of the current training sample is input into the corresponding pre-trained encoder to extract the multimodal feature vector of the current training sample. The key vector set and value vector set of historical wafer defect cases are extracted from the database. An attention weight between the multimodal feature vector and the key vector set of the current training sample is calculated using a cross-attention mechanism. Based on the attention weight, the value vector set is weighted and aggregated. The prior knowledge of the most relevant historical wafer defect cases is integrated into the multimodal feature vector of the current training sample, generating a multimodal enhanced query vector for the current training sample. This constructs a defect knowledge base with "long-term memory" capabilities, realizing a closed-loop feedback structure of "memory guidance - context enhancement." By introducing prior knowledge of historical wafer defect cases, the retrieval model's representation ability in small sample scenarios is significantly improved, enabling the retrieval model to learn richer defect patterns from limited training samples and improving the retrieval accuracy of historical wafer defect cases.

[0053] Furthermore, the augmented query vectors of different modalities in the current training sample are divided into time steps to obtain the process stages corresponding to the augmented query vectors of different modalities in the current training sample. According to the time sequence of the process flow, the augmented query vectors of each modality at the same time step are alternately concatenated to form a multimodal sequence of the current training sample. The multimodal sequence is input into a sequence processing module based on a state-space mechanism. The sequence processing module performs temporal modeling of the multimodal sequence through a state-space model architecture. During the forward propagation process, the sequence processing module introduces an optimal transmission strategy to perform fine-grained alignment of the hidden states of different modalities. The sequence processing module outputs the fusion features of different modalities of the current training sample after cross-modal alignment optimization. In this way, the shortcomings of existing methods that ignore the consistency of the time dimension are solved, the dynamic correlation in the defect evolution process is effectively captured, and the retrieval model can understand the evolution law of defects in the process flow, thereby improving the defect root cause reasoning ability across process stages.

[0054] Furthermore, the first maximum mean difference between the semantic enhancement features of the defect image modality and the context-aware features of the defect description text modality of the current training sample in the feature space is calculated. The second maximum mean difference between the temporal fusion features of the equipment process parameter modality and the context-aware features of the defect description text modality of the current training sample in the feature space is also calculated. The first maximum mean difference and the second maximum mean difference are weighted and summed to form the global alignment loss function. The global alignment loss function is minimized through the backpropagation algorithm to optimize the parameters of the cross-attention module, vector concatenation module and sequence processing module until the model converges. This enables the features of each modality to achieve distribution consistency in a unified feature space, effectively suppressing semantic drift caused by modal heterogeneity under small sample conditions, and ensuring the semantic alignment accuracy of defect images, text descriptions and process parameters on the evolution path, thereby improving the cross-modal retrieval robustness and generalization ability of the retrieval model. Attached Figure Description

[0055] The above and other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0056] Figure 1 The diagram shown is a schematic framework of an exemplary training method for a retrieval model for wafer defect cases according to an embodiment of this application.

[0057] Figure 2 The diagram shown is a schematic flowchart of an exemplary training method for a retrieval model for wafer defect cases according to an embodiment of this application.

[0058] Figure 3 The diagram shown is a schematic framework of an exemplary method for retrieving wafer defect cases according to an embodiment of this application.

[0059] Figure 4 The diagram shown is an exemplary flowchart of a method for retrieving wafer defect cases according to an embodiment of this application.

[0060] Figure 5 The diagram shown is a schematic structural diagram of an exemplary training apparatus for a retrieval model of wafer defect cases according to an embodiment of this application.

[0061] Figure 6 The diagram shown is a schematic structural representation of an exemplary device for retrieving wafer defect cases according to an embodiment of this application.

[0062] Figure 7 The diagram shown is a schematic representation of an exemplary electronic device according to an embodiment of this application. Detailed Implementation

[0063] The present application will now be described in more detail with reference to the accompanying drawings. In the various drawings, the same elements are indicated by similar reference numerals. For clarity, the various parts in the drawings are not drawn to scale. Furthermore, some well-known parts may not be shown.

[0064] This application may be presented in various forms, some of which will be described below.

[0065] Figure 1 The diagram shown is a schematic framework of an exemplary training method for a retrieval model of wafer defect cases according to an embodiment of this application. Figure 1 As shown, the overall framework of the training method for the retrieval model 101 for wafer defect cases includes a database 140, an image encoder 150, a text encoder 160, and the retrieval model 101. The database 140 stores data units formed by encoding multimodal raw data of historical wafer defect cases. Each data unit contains a key vector (Kmemory) and a value vector (Vmemory), used to enable dynamic retrieval and incremental updates of historical knowledge.

[0066] The retrieval model 101 includes a cross-attention module 110, a vector concatenation module 120, and a sequence processing module 130. During training, the multimodal raw data X1 of the current training sample (including defect images and defect description text) is processed by the image encoder 150 and the text encoder 160 to extract high-dimensional features, resulting in the image modality feature vector Xm1 and the text modality feature vector Xt1 of the current training sample. The cross-attention module 110 receives the feature vectors Xm1 and Xt1 of the current training sample and extracts the key vector set Kmemory and the value vector set Vmemory of historical cases from the database 140. It calculates the attention weights through the cross-attention mechanism, performs weighted aggregation on the value vector set Vmemory, and integrates the prior knowledge of the most relevant historical wafer defect cases into the multimodal feature vector of the current training sample to generate the multimodal enhanced query vector Mquery1 of the current training sample.

[0067] The vector concatenation module 120 adopts a "time step priority" concatenation strategy, aligning and concatenating the enhanced query vectors Mquery1 of different modalities of the current training sample according to the time sequence of the process flow to form a unified multimodal sequence Xconcat1. This concatenation strategy takes the time axis as the main line, aligning image, text and other modal features at each time point and concatenating them sequentially to ensure the consistency of cross-modal information in the time dimension and effectively capture the dynamic correlation in the defect evolution process.

[0068] After receiving the multimodal sequence Xconcat1 of the current training sample, the sequence processing module 130 extracts the fusion features of different modalities of the current training sample. In some embodiments, the sequence processing module 130 is based on a state-space model (SSM) architecture, which has the ability to efficiently process ultra-long sequences and can model long-range dependencies across time steps. The sequence processing module 130 internally introduces an optimal transport (OT) strategy to perform fine-grained alignment of the hidden states of different modalities during forward propagation, and outputs the fusion features of different modalities of the current training sample after cross-modal alignment optimization, including semantic enhancement features Xa1 of the defect image modality, temporal fusion features Xb1 of the equipment process parameter modality, and context-aware features Xc1 of the defect description text modality.

[0069] In addition, the distribution distance of the fusion features of different modalities of the current training sample in the same feature space is calculated, a global alignment loss function Lalign is constructed, and the parameters of the retrieval model 101 are optimized through the backpropagation algorithm to ensure that heterogeneous modalities such as images and text converge toward a unified language anchor distribution, thereby achieving global semantic consistency.

[0070] Figure 2 The diagram shown is a schematic flowchart illustrating an exemplary method for training a retrieval model for wafer defect cases according to an embodiment of this application. Figure 2 As shown, the training methods include:

[0071] In step S210, the prior knowledge of historical wafer defect cases in the database is incorporated into the multimodal feature vector of the current training sample through the cross-attention module to generate the multimodal enhanced query vector of the current training sample.

[0072] In step S220, the vector splicing module splices the enhanced query vectors of different modalities of the current training sample into a multimodal sequence of the current training sample according to the time sequence of the process flow.

[0073] In step S230, the multimodal sequence of the current training sample is input into the sequence processing module to extract the fusion features of different modalities of the current training sample.

[0074] In step S240, the distribution distance of the fusion features of different modalities of the current training sample in the same feature space is calculated, thereby constructing a global alignment loss function, and optimizing the parameters of the retrieval model through the backpropagation algorithm.

[0075] The following is combined Figure 1 right Figure 2 The training method of the retrieval model 101 shown will be described in detail.

[0076] In some embodiments, before step S210, database 140 needs to be established and initialized. Specifically, multimodal raw data of historical wafer defect cases (including defect images, text descriptions, process parameters, equipment logs, etc.) are input into corresponding pre-trained encoders (such as image encoder 150 and text encoder 160), their high-dimensional feature vectors are extracted, and encoded into searchable data units and stored in database 140. Each data unit contains two core components: a key vector Kmemory and a value vector Vmemory. The key vector Kmemory is used for similarity matching and is usually a lightweight encoded semantic embedding; the value vector Vmemory contains complete contextual information and is used for knowledge retrieval and enhancement. All data units are indexed by the timestamp of the defect occurrence, supporting incremental updates and time- or importance-based aging and elimination mechanisms to ensure that database 140 continuously reflects the latest status of the production line and defect distribution trends, forming a defect knowledge base with "long-term memory" capabilities, and realizing a closed-loop feedback structure of "memory guidance - context enhancement".

[0077] During the initialization of database 140, historical wafer defect cases are preprocessed. The multimodal data of each case is converted into feature vectors using a corresponding encoder, and these features are organized according to the time sequence of the process flow. In database 140, for each historical wafer defect case, its multimodal feature vectors are extracted, such as image features, text features, and process parameter features. These features are then aligned according to the corresponding process time steps to form time-series data units. These data units are stored in database 140, providing a historical knowledge foundation for the subsequent cross-attention mechanism. Furthermore, database 140 is indexed and optimized to establish an efficient retrieval mechanism, ensuring that the most relevant historical cases to the current sample can be quickly located during model training and retrieval. This timestamp-based index structure not only supports chronological retrieval but also allows the system to filter based on multiple dimensions such as defect type and process stage, improving the accuracy and efficiency of historical knowledge retrieval.

[0078] In step S210, during the training process, the multimodal raw data X1 of the current training sample (including defect images and defect description text) is input into the corresponding pre-trained encoders (e.g., image encoder 150 and text encoder 160) to extract the multimodal feature vectors of the current training sample (e.g., image modal feature vector Xm1 and text modal feature vector Xt1). The key vector Kmemory set and value vector Vmemory set of historical wafer defect cases are extracted from the database 140. The multimodal feature vectors of the current training sample (e.g., image modal feature vector Xm1 and text modal feature vector Xt1) and the key vector Kmemory set and value vector Vmemory set of historical wafer defect cases are input into the cross-attention module 110. The attention weights between the multimodal feature vectors of the current training sample and the key vector Kmemory set are calculated through the cross-attention mechanism. Based on the attention weights, the value vector Vmemory set is weighted and aggregated, incorporating the prior knowledge of the most relevant historical wafer defect cases into the multimodal feature vectors of the current training sample to generate the multimodal enhanced query vector Mquery1 of the current training sample. The multimodal enhanced query vector Mquery1 of the current training sample may include image modal enhanced query vectors and text modal enhanced query vectors. In some embodiments, the calculation formula for the multimodal enhanced query vector Mquery1 of the current training sample is:

[0079] Mquery1=CrossAttention(Q,Kmemory,Vmemory)(1)

[0080] Here, Q represents the set of multimodal feature vectors of the current training sample, Kmemory and Vmemory correspond to the key vector and value vector sets of historical wafer defect cases indexed by timestamp in the database 140, respectively; the key vector Kmemory is used to measure its semantic relevance to the set of multimodal feature vectors Q of the current training sample, while the value vector Vmemory carries the complete contextual information of the corresponding historical wafer defect case; CrossAttention is, for example, a standard cross-attention function, which calculates the attention distribution by scaling the dot product to ensure the stability and interpretability of historical knowledge transfer.

[0081] Understandably, the cross-attention module 110 enables dynamic retrieval and incremental updates of historical knowledge, allowing current training samples to acquire prior knowledge from historical cases. This significantly improves the retrieval model 101's ability to identify rare defect patterns, especially in small-sample scenarios. By introducing contextual information from historical cases, the model's representational ability in such scenarios is significantly enhanced, enabling it to learn richer defect patterns from limited training samples and improving the retrieval accuracy of historical wafer defect cases. This effectively compensates for the lack of training data and enhances the model's generalization performance.

[0082] In step S220, the multimodal augmented query vector Mquery1 of the current training sample is input into the vector concatenation module 120. The vector concatenation module 120 divides the augmented query vectors Mquery1 of different modalities of the current training sample into time steps to obtain the process stages corresponding to the augmented query vectors Mquery1 of different modalities of the current training sample. According to the time sequence of the process flow, the augmented query vectors of each modality at the same time step are alternately concatenated to form the multimodal sequence Xconcat1 of the current training sample. In some embodiments, the calculation formula of the multimodal sequence Xconcat1 of the current training sample is:

[0083] (2)

[0084] in, It is the image modality-enhanced query vector at the i-th time step. It is the text modality-enhanced query vector for the i-th time step; i is greater than 0 and less than or equal to T, i and M are integers, and M is the total number of process time steps in the process flow. According to formula (2), the image modality-enhanced query vector for the i-th time step is... and the text modality-enhanced query vector at the i-th time step Alternately concatenate the vectors to generate the concatenated enhanced query vector at the i-th time step. Based on the time sequence of the process flow, multiple time steps in the process flow are concatenated to enhance the query vector. Sequential concatenation is performed to generate the multimodal sequence Xconcat1 of the current training samples. This time-aligned concatenation strategy not only preserves the temporal structure of each modality-enhanced query vector but also strengthens the coupling relationship between images and text along the process evolution path through alternating arrangement, enabling the defect causes and characterization changes to form a traceable causal chain in a unified sequence. This "time-step-first" concatenation strategy ensures the consistency of cross-modal information in the time dimension, allowing the model to accurately capture the evolutionary patterns of defects at different process stages. It effectively solves the problem of inaccurate defect evolution path analysis caused by time dimension misalignment in traditional methods, providing a reliable data foundation for subsequent temporal modeling and root cause analysis.

[0085] In step S230, the multimodal sequence Xconcat1 of the current training sample is input into the sequence processing module 130 based on a state-space mechanism. The sequence processing module 130 performs temporal modeling of the multimodal sequence Xconcat1 using a state-space model architecture. During forward propagation, the sequence processing module 130 introduces an optimal transmission strategy to perform fine-grained alignment of the hidden states of different modalities. The sequence processing module 130 outputs the fusion features of different modalities of the current training sample after cross-modal alignment optimization. The fusion features of different modalities of the current training sample may include semantic enhancement features Xa1 of the defect image modality, temporal fusion features Xb1 of the equipment process parameter modality, and context-aware features Xc1 of the defect description text modality. This addresses the shortcomings of existing methods that neglect temporal consistency, effectively captures the dynamic correlations in the defect evolution process, enables the retrieval model to understand the evolutionary patterns of defects in the process flow, and thus improves the ability to infer the root causes of defects across process stages.

[0086] In step S240, the first maximum mean difference (MMD) is calculated between the semantic enhancement features Xa1 of the defect image modality and the context-aware features Xc1 of the defect description text modality of the current training samples in the feature space (e.g., the reproducing kernel Hilbert space (RKHS)). 2 (Xa1, Xc1); Calculate the second maximum mean difference (MMD) between the temporal fusion feature Xb1 of the equipment process parameter modality and the context-aware feature Xc1 of the defect description text modality of the current training sample in the feature space. 2 (Xb1,Xc1); the first maximum mean difference (MMD) 2 (Xa1,Xc1) and the second largest mean difference (MMD) 2 The weighted sum of (Xb1, Xc1) constitutes the global alignment loss function Lalign. In some embodiments, the formula for calculating the global alignment loss function Lalign is:

[0087] Lalign=α×MMD²(Xa1,Xc1)+β×MMD 2 (Xb1,Xc1)(3)

[0088] Wherein, α and β are learnable weight coefficients, which respectively adjust the contribution of image modality and process parameter modality to the text modality anchor point alignment, ensuring that each modality converges collaboratively in a unified semantic space; MMD 2 (Xa1, Xc1) is a measure of the distribution difference between image modal semantic enhancement features Xa1 and text modal context-aware features Xc1 in the feature space, MMD. 2 (Xb1, Xc1) is the distribution difference measure between the equipment process parameter modal temporal fusion feature Xb1 and the text modal context-aware feature Xc1; the first maximum mean difference (MMD) 2 The smaller (Xa1,Xc1) is, the more consistent the semantic distribution of the image modal semantic enhancement feature Xa1 and the text modal context-aware feature Xc1 are; the second largest mean difference (MMD) 2 The smaller (Xb1, Xc1) is, the closer the semantic distribution of the equipment process parameter modal temporal fusion feature Xb1 and the text modal context-aware feature Xc1 is to the unified representation space; when both converge to below the threshold, the model completes cross-modal semantic anchoring. Since calculating the maximum mean difference (MMD) based on the Gaussian kernel function is a commonly used method in related technologies, the process of calculating the maximum mean difference will not be elaborated here.

[0089] In some embodiments, the parameters of the cross-attention module 110, vector concatenation module 120, and sequence processing module 130 are optimized by minimizing the global alignment loss function Lalign through backpropagation until the model converges. It is understood that the modal features achieve consistent distribution in a unified feature space, effectively suppressing semantic drift caused by modal heterogeneity in small sample sizes, ensuring the semantic alignment accuracy of defect images, text descriptions, and process parameters along the evolution path, thereby improving the cross-modal retrieval robustness and generalization ability of the retrieval model.

[0090] Figure 3 The diagram shown is a schematic framework of an exemplary method for retrieving wafer defect cases according to an embodiment of this application. Figure 3 As shown, the overall framework 300 for the retrieval method for wafer defect cases includes a database 140, an image encoder 150, a text encoder 160, a retrieval model 101 trained using the above training method, and a similarity matching module 170. Figure 3 The frame 300 shown is Figure 1 The frame 100 shown is basically similar, and for convenience, the similarities will not be described again here.

[0091] In the application phase of retrieval model 101, the user inputs the multimodal raw data (including defect images and defect description text) of the wafer defect cases to be retrieved into the pre-trained encoder (including image encoder 150 and text encoder 160) to obtain the image modal feature vector Xm2 and text modal feature vector Xt2 of the wafer defect cases to be retrieved. The cross-attention module 110 receives the image modal feature vector Xm2 and text modal feature vector Xt2 of the wafer defect cases to be retrieved, and extracts the key vector set Kmemory and value vector set Vmemory of historical cases from the database 140. Through the cross-attention mechanism, attention weights are calculated, and the value vector set Vmemory is weighted and aggregated. The prior knowledge of the most relevant historical wafer defect cases is integrated into the multimodal feature vector of the current training sample to generate the multimodal enhanced query vector Mquery2 of the wafer defect cases to be retrieved.

[0092] The vector stitching module 120 employs a "time step priority" stitching strategy, aligning and stitching the enhanced query vectors Mquery2 of different modalities of the wafer defect cases to be retrieved according to the time sequence of the process flow, forming a unified multimodal sequence Xconcat2. This stitching strategy uses the time axis as the main line, aligning image, text, and other modal features at each time point before stitching them sequentially, ensuring the consistency of cross-modal information in the time dimension and effectively capturing the dynamic correlations in the defect evolution process.

[0093] After receiving the multimodal sequence Xconcat2 of the wafer defect case to be retrieved, the sequence processing module 130 extracts the fusion features of different modes of the wafer defect case to be retrieved. In some embodiments, the sequence processing module 130 is based on a state-space model (SSM) architecture, which has the ability to efficiently process ultra-long sequences and can model long-range dependencies across time steps. The sequence processing module 130 internally introduces an optimal transport (OT) strategy to perform fine-grained alignment of the hidden states of different modes during forward propagation, and outputs the fusion features of different modes of the wafer defect case to be retrieved after cross-modal alignment optimization, including semantic enhancement features Xa2 of the defect image mode, temporal fusion features Xb2 of the equipment process parameter mode, and context-aware features Xc2 of the defect description text mode.

[0094] In addition, the similarity matching module 170 receives the fusion features output by the sequence processing module 130, and calculates the similarity between the fusion features of the wafer defect case to be retrieved and the feature vectors of historical wafer defect cases in the database 140 based on the similarity measurement algorithm. According to the similarity matching sorting order, it returns one or more historical wafer defect cases with the highest similarity to the user.

[0095] Figure 4The diagram shown is an exemplary flowchart of a method for retrieving wafer defect cases according to an embodiment of this application. Figure 4 The retrieval method shown is applied to Figure 3 The retrieval model 101 shown is as follows. Figure 4 As shown, the retrieval methods include:

[0096] In step S410, the multimodal feature vector of the wafer defect case to be retrieved is input into the trained retrieval model. Through the cross-attention module, the prior knowledge of historical wafer defect cases in the database is integrated into the multimodal feature vector of the wafer defect case to be retrieved, thereby generating the multimodal enhanced query vector of the wafer defect case to be retrieved.

[0097] In step S420, the enhanced query vectors of different modes of the wafer defect cases to be retrieved are concatenated into a multimodal sequence of the wafer defect cases to be retrieved according to the time sequence of the process flow through the vector concatenation module.

[0098] In step S430, the multimodal sequence of the wafer defect case to be retrieved is input into the sequence processing module to extract the fusion features of different modes of the wafer defect case to be retrieved.

[0099] In step S440, the fusion features of different modes of the wafer defect case to be retrieved are used to perform similarity matching and sorting with the multimodal fusion features of historical wafer defect cases in the database, and one or more historical wafer defect cases with the highest similarity are returned as the retrieval results.

[0100] Due to the use of Figure 3 The specific process of the retrieval method for wafer defect cases implemented by the retrieval model 101 shown has been described in detail above and will not be repeated here.

[0101] Figure 5 The diagram shows a schematic representation of an exemplary training apparatus for a wafer defect case retrieval model according to an embodiment of this application. The retrieval model includes a cross-attention module, a vector concatenation module, and a sequence processing module. Figure 5 As shown, the training device 500 includes a first vector enhancement unit 510, a first vector splicing unit 520, a first vector fusion unit 530, and a parameter optimization unit 540.

[0102] The first vector enhancement unit 510 is used to integrate prior knowledge of historical wafer defect cases in the database into the multimodal feature vector of the current training sample through the cross-attention module, and generate the multimodal enhanced query vector of the current training sample.

[0103] The first vector splicing unit 520 is used to splice the enhanced query vectors of different modalities of the current training sample into a multimodal sequence of the current training sample according to the time sequence of the process flow through the vector splicing module.

[0104] The first vector fusion unit 530 is used to input the multimodal sequence of the current training sample into the sequence processing module and extract the fusion features of different modes of the current training sample.

[0105] The parameter optimization unit 540 is used to calculate the distribution distance of the fusion features of different modalities of the current training sample in the same feature space, thereby constructing a global alignment loss function and optimizing the parameters of the retrieval model through the backpropagation algorithm.

[0106] Since the specific process of model training has been described in detail above, it will not be repeated here.

[0107] Figure 6 The diagram shows an exemplary retrieval device for wafer defect cases according to an embodiment of this application. The retrieval device 600 of this embodiment is applied to a retrieval model trained according to the above-described training method. As shown in Figure 6, the retrieval device 600 includes a second vector enhancement unit 610, a second vector splicing unit 620, a second vector fusion unit 630, and a similarity matching unit 640.

[0108] The second vector enhancement unit 610 is used to input the multimodal feature vector of the wafer defect case to be retrieved into the trained retrieval model, and through the cross attention module, integrate the prior knowledge of historical wafer defect cases in the database into the multimodal feature vector of the wafer defect case to be retrieved, thereby generating the multimodal enhanced query vector of the wafer defect case to be retrieved.

[0109] The second vector splicing unit 620 is used to splice the enhanced query vectors of different modes of the wafer defect case to be retrieved into a multimodal sequence of the wafer defect case to be retrieved in the order of the process flow through the vector splicing module.

[0110] The second vector fusion unit 630 is used to input the multimodal sequence of the wafer defect case to be retrieved into the sequence processing module and extract the fusion features of different modes of the wafer defect case to be retrieved.

[0111] The similarity matching unit 640 is used to perform similarity matching and sorting by using the fusion features of different modes of the wafer defect case to be retrieved and the multimodal fusion features of historical wafer defect cases in the database, and return one or more historical wafer defect cases with the highest similarity as the retrieval results.

[0112] Due to the use of Figure 3The specific process of the retrieval method for wafer defect cases implemented by the retrieval model 101 shown has been described in detail above and will not be repeated here.

[0113] This disclosure also provides an electronic device 700, such as... Figure 7 As shown, it includes a memory 720, a processor 710, a power supply component 730, a network interface 740, an input / output interface 750, and a program stored in the memory 720 and executable on the processor 710. When the program is executed by the processor 710, it can implement the various processes of the embodiments of the above methods and achieve the same technical effects. To avoid repetition, it will not be described again here.

[0114] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, this disclosure also provides a storage medium storing a computer program or instructions that, when executed by a processor, can implement the various processes of the embodiments of the above methods.

[0115] Since the instructions stored in the storage medium can execute the steps of the method provided in the embodiments of this disclosure, the beneficial effects achievable by the method provided in the embodiments of this disclosure can be realized, as detailed in the preceding embodiments, and will not be repeated here. Specific implementations of the above operations can be found in the preceding embodiments, and will not be repeated here.

[0116] Finally, it should be noted that the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The embodiments described above, as per the implementation of this application, do not exhaustively describe all details, nor do they limit the application to only the specific embodiments described. Clearly, many modifications and variations can be made based on the above description. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of this application, thereby enabling those skilled in the art to make good use of this application and modifications based on it. This application is limited only by the claims and their full scope and equivalents.

Claims

1. A training method for a retrieval model for wafer defect cases, the retrieval model comprising a cross-attention module, a vector concatenation module, and a sequence processing module, the training method comprising: The cross-attention module integrates prior knowledge of historical wafer defect cases in the database into the multimodal feature vector of the current training sample to generate the multimodal enhanced query vector of the current training sample. The vector concatenation module concatenates the enhanced query vectors of different modalities of the current training sample into a multimodal sequence of the current training sample according to the time sequence of the process flow. The multimodal sequence of the current training sample is input into the sequence processing module to extract the fusion features of different modalities of the current training sample; The distribution distance of the fusion features of different modalities of the current training sample in the same feature space is calculated to construct a global alignment loss function, and the parameters of the retrieval model are optimized by the backpropagation algorithm.

2. The training method according to claim 1, wherein, Before generating the multimodal enhanced query vector of the current training sample by incorporating prior knowledge of historical wafer defect cases from the database into the multimodal feature vector of the current training sample through the cross-attention module, the training method further includes: The database is constructed, which includes multiple data units. Each data unit is formed by encoding multimodal raw data of historical wafer defect cases, and each data unit includes a key vector and a value vector.

3. The training method according to claim 2, wherein, In the database, for each historical wafer defect case, its multimodal feature vector is extracted, and these feature vectors are aligned according to the corresponding process time steps to form time-series data units.

4. The training method according to claim 3, wherein, The process of integrating prior knowledge of historical wafer defect cases from the database into the multimodal feature vector of the current training sample through the cross-attention module to generate the multimodal enhanced query vector of the current training sample includes: The multimodal raw data of the current training sample are input into the corresponding pre-trained encoder to extract the multimodal feature vector of the current training sample; Extract the set of key vectors and the set of value vectors for historical wafer defect cases from the database; The attention weight between the multimodal feature vector of the current training sample and the key vector set is calculated using a cross-attention mechanism. The value vector set is weighted and aggregated based on the attention weights, and the prior knowledge of the most relevant historical wafer defect cases is incorporated into the multimodal feature vector of the current training sample to generate the multimodal enhanced query vector of the current training sample.

5. The training method according to claim 3, wherein, The step of using the vector concatenation module to concatenate the enhanced query vectors of different modalities of the current training sample into a multimodal sequence of the current training sample according to the time sequence of the process flow includes: The enhanced query vectors of different modalities of the current training sample are divided into time steps to obtain the process stages corresponding to the enhanced query vectors of different modalities of the current training sample. Based on the time sequence of the process flow, the modal augmented query vectors at the same time step are alternately concatenated to form the multimodal sequence of the current training sample.

6. The training method according to claim 5, wherein, The enhanced query vectors of different modalities of the current training sample include image modal enhanced query vectors and text modal enhanced query vectors. The step of alternately concatenating the enhanced query vectors of each modality at the same time step according to the time sequence of the process flow to form the multimodal sequence of the current training sample includes: The image modal augmented query vector and the text modal augmented query vector at the i-th time step are alternately concatenated to generate the concatenated augmented query vector at the i-th time step, where i is an integer greater than 0 and less than or equal to the total number of process time steps in the process flow; Based on the time sequence of the process flow, the concatenation enhancement query vectors of multiple time steps in the process flow are concatenated sequentially to generate the multimodal sequence of the current training sample.

7. The training method according to claim 3, wherein, The step of inputting the multimodal sequence of the current training sample into the sequence processing module and extracting the fusion features of different modalities of the current training sample includes: The multimodal sequence is input into the sequence processing module based on the state-space mechanism, and the sequence processing module performs time-series modeling on the multimodal sequence through the state-space model architecture; During forward propagation, the sequence processing module introduces an optimal transmission strategy to perform fine-grained alignment of the hidden states of different modalities. The sequence processing module outputs the fusion features of different modalities of the current training sample after cross-modal alignment optimization.

8. The training method according to claim 3, wherein, The fusion features of different modalities of the current training sample include semantic enhancement features of the defect image modality, temporal fusion features of the equipment process parameter modality, and context-aware features of the defect description text modality. The calculation of the distribution distance of the fusion features of different modalities of the current training sample in the same feature space, thereby constructing a global alignment loss function, and optimizing the parameters of the wafer defect case retrieval model through a backpropagation algorithm, includes: Calculate the first maximum mean difference between the semantic enhancement features of the defect image modality and the context-aware features of the defect description text modality of the current training sample in the feature space; Calculate the second maximum mean difference between the temporal fusion features of the equipment process parameter modality and the context-aware features of the defect description text modality of the current training sample in the feature space; The first maximum mean difference and the second maximum mean difference are weighted and summed to form the global alignment loss function; The global alignment loss function is minimized using the backpropagation algorithm, and the parameters of the cross-attention module, the vector concatenation module, and the sequence processing module are optimized until the model converges.

9. A method for retrieving wafer defect cases, applied to a retrieval model trained by the training method according to any one of claims 1 to 8, the retrieval method comprising: The multimodal feature vector of the wafer defect case to be retrieved is input into the trained retrieval model. Through the cross attention module, the prior knowledge of historical wafer defect cases in the database is integrated into the multimodal feature vector of the wafer defect case to be retrieved, thereby generating the multimodal enhanced query vector of the wafer defect case to be retrieved. The vector splicing module splices the enhanced query vectors of different modes of the wafer defect cases to be retrieved into a multimodal sequence of the wafer defect cases to be retrieved according to the time sequence of the process flow. The multimodal sequence of the wafer defect case to be retrieved is input into the sequence processing module to extract the fusion features of different modes of the wafer defect case to be retrieved; The fusion features of different modes of the wafer defect case to be retrieved are used to perform similarity matching and sorting with the multimodal fusion features of historical wafer defect cases in the database, and one or more historical wafer defect cases with the highest similarity are returned as the retrieval results.

10. An electronic device, comprising: One or more processors; A memory for storing executable instructions, which, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1-9.

Images