A knowledge reasoning method, device and medium for an integrated circuit wafer process

Abstract

The application belongs to the field of wafer manufacturing, and discloses a knowledge reasoning method, device and medium for an integrated circuit wafer process, which comprises the following steps: extracting multi-modal features from a wafer image; performing wafer knowledge retrieval; performing document sorting; and generating a query answer. The application gradually realizes dynamic decomposition and high-precision solution of a complex query through three core stages. The introduced multi-modal chain reasoning framework enables the model to combine logical reasoning and multi-modal information, and constructs a coherent solution chain to cope with complex tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of wafer manufacturing, and particularly to a knowledge reasoning method, device and medium for integrated circuit wafer process. BACKGROUND

[0002] Wafer manufacturing is a critical process for processing high-purity silicon wafers into semiconductor chips. As a core step in integrated circuit (IC) production, wafer manufacturing directly affects the performance, power consumption and reliability of chips. In this process, wafers need to go through precise processes such as photolithography, ion implantation, thin film deposition and etching to build complex microcircuits. However, even minor cracks, particle contamination or uneven oxide layers on the surface can significantly affect the performance and yield of integrated circuits, and even cause device failure. As chip manufacturing processes evolve towards smaller nodes (such as 3nm and below), quality control in wafer manufacturing faces higher technical challenges, and integrated, automated wafer defect detection, root cause analysis and reasoning query technologies are crucial for improving semiconductor manufacturing yield and driving technological progress.

[0003] In recent years, breakthroughs in large language models (LLMs) in natural language processing (NLP) have significantly improved the ability of artificial intelligence (AI) in language understanding and generation. Multimodal large language models (MLLMs) integrate language, visual and audio information, providing new solutions for defect detection and root cause analysis in wafer analysis. For example, by combining optical microscopy images and detection log data, MLLMs can help engineers quickly locate the root cause of wafer defects. However, these models still face the challenge of "hallucinations" (generating inaccurate or false information) when applied to the field of integrated circuits, and there are significant limitations in understanding the highly specialized domain knowledge of wafer manufacturing.

[0004] To address these issues, methods based on the Retrieval Augmented Generation (RAG) framework have been widely adopted. The RAG framework improves the accuracy and domain adaptability of generation by retrieving relevant information from external knowledge bases and integrating it into the model input. In the wafer manufacturing field, RAG technology can utilize structured documents (such as manufacturing process standards, defect analysis reports) to optimize the model's understanding of process steps and terminology. Retrieval strategies are generally divided into three categories: sparse retrieval (such as keyword search, such as BM25, TF-IDF, and inverted index) relies on the exact matching of query and document vocabulary, with high computational efficiency. In structured wafer knowledge documents with fixed terminology, it works well, but it is difficult to capture the semantic similarity between words, thus limiting the understanding of rich terminology in wafer process and defect analysis. Dense retrieval (semantic search) uses embedding models to convert text into high-dimensional vectors, identifying relevant documents through vector similarity, and can capture deep semantic relationships in wafer manufacturing processes, but has high computational cost. Hybrid retrieval combines and deduplicates the results of the two retrieval methods, balancing efficiency and semantic understanding. These strategies enhance the information retrieval capabilities of RAG and the reasoning capabilities of large models in specialized fields to some extent.

[0005] Despite these strategies being effective, there are still the following limitations: 1) insufficient understanding of professional wafer knowledge: wafer manufacturing involves complex processes and specialized knowledge, but existing models have difficulty understanding some uncommon terms (such as "doping" and "oxidation") in general databases, exacerbating the hallucination phenomenon; 2) omission of details in complex queries: for direct document retrieval of complex queries (such as defect root cause analysis or surface treatment processes), it is difficult to cover all key information, especially when multiple technical details are involved, important content may be missed, affecting the integrity of reasoning; 3) interference of redundant or irrelevant documents: the documents returned by existing retrieval strategies are often lengthy and contain a lot of irrelevant information, which may introduce text noise and weaken the reliability of the generation model.

[0006] Therefore, the present invention aims to solve the following three technical problems:

[0007] 1) How to optimize retrieval augmented generation to enhance comprehensive understanding of complex professional tasks and effectively alleviate the occurrence of hallucination problems;

[0008] 2) How to efficiently analyze the query intent and retrieve wafer knowledge in detail to improve the authenticity and accuracy of the language model's answers;

[0009] 3) How to provide the model with more reliable document knowledge while reducing the computational burden of the model and reducing the interference of irrelevant knowledge. SUMMARY

[0010] The present application aims to provide a wafer process knowledge reasoning method, device and medium to solve the problems in the background art.

[0011] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0012] A wafer process knowledge reasoning method, comprising:

[0013] Step 1, through the extraction of multi-modal features of the input wafer image query, the image surface defect detection is performed, and finally the Mask image capable of intuitively presenting the defect area is generated;

[0014] Step 2, wafer knowledge retrieval is performed, in which the retrieval task is divided into multiple sub-steps according to the complexity of the user query, and in each sub-step, the wafer image, defect mask information and wafer ID are dynamically integrated to gradually complete the search of related documents;

[0015] Step 3, document sorting, in which the retrieval results of wafer knowledge retrieval are weighted and scored, and the retrieved documents are prioritized through a scientific scoring mechanism, and the top several documents most relevant to the query are selected;

[0016] Step 4, generating query answers, in which the wafer image, defect mask and sorted high-relevance documents are input as prompts into the basic language model, and the basic language model generates accurate and explanatory answers by combining multi-modal data and professional document information.

[0017] Further, the step 1 comprises: using a frozen pre-training encoder to encode the input wafer image and the extracted wafer ID information, and fusing the visual features and text features into a multi-modal vector representation.

[0018] Further, the step 2 comprises:

[0019] Step 2.1, logical decomposition of query: design a query reasoner to decompose the user query into multiple sub-queries and form a solution chain to capture key details, and in this stage, the basic language model can recognize whether the query needs to be decomposed and generate corresponding sub-queries through prompt learning to fine-tune the data set of different complexity queries;

[0020] For simple queries, the final answer is directly output; for complex queries, the length of the solution chain is dynamically adjusted to further optimize the logical plan;

[0021] Step 2.2, cross-modal fusion: a shared gating unit is designed to evaluate the relevance of wafer images, generated masks, and extracted wafer ID information to each sub-query, and the most relevant modal information is selected for fusion with the query vector.

[0022] Step 2.3, multi-round retrieval: in the multi-round retrieval stage, based on logical decomposition and cross-modal fusion, relevant candidate documents are retrieved from the wafer knowledge base for each user query and its sub-queries and multi-modal vectors; each round of retrieval provides necessary external knowledge hints for generating intermediate answers; a hybrid retrieval method is used to combine keyword-based retrieval and semantic-based retrieval; in operation, sub-queries use keyword matching to ensure precise alignment of terms, and multi-modal vectors capture complex contextual relationships through semantic retrieval.

[0023] Further, the step 2.2 specifically includes:

[0024] The frozen pre-trained encoder generates picture vectors, mask vectors, and text vectors, which are concatenated and mapped to a smaller hidden space through linear transformation, thereby achieving comprehensive evaluation of multi-modal information;

[0025] Subsequently, the relevance between modalities is calculated through self-attention mechanism, and then Softmax is used to assign weights to filter out the most relevant modal information for subsequent reasoning. Finally, a multi-modal query vector is obtained.

[0026] Further, the step 3 specifically includes:

[0027] Step 3.1, multi-dimensional scoring strategy for candidate documents: for each sub-query and its multi-modal query vector, three retrievers, TF-IDF, BM25, and Faiss, are used to extract candidate document sets from the knowledge base, and finally each document is assigned a corresponding score;

[0028] Step 3.2, semantic density calculation and retriever weight adaptive adjustment: calculate the semantic density of each sub-query;

[0029] Based on the semantic density, the weight vectors of TF-IDF, BM25, and Faiss are adaptively adjusted to achieve dynamic balance of scoring. The final score is calculated based on the scores and weight vectors of TF-IDF, BM25, and Faiss, the semantic density, the balance coefficient, and the final score.

[0030] Step 3.3, accurate reordering of candidate documents: in the final sorting stage, the candidate documents are reordered according to the final score, and the most relevant documents are selected from them.

[0031] Further, the step 4 includes:

[0032] The LoRA technology is used to fine-tune the basic language model, and the basic language model is customized into a special multi-modal large language model based on the training of the multi-modal wafer corpus. In the fine-tuning process, the input mode is expanded according to the characteristics of the wafer in the integrated circuit field:

[0033] Firstly, the mask vector extracted from the query image is taken as the visual input to enhance the accurate understanding of the model to the defect area;

[0034] Secondly, the top several knowledge documents retrieved in relation to the query are attached to the user query as additional text input to provide rich context support for the model;

[0035] In the model adaptation process, the LoRA technology adjusts the model parameters by introducing a low-rank matrix.

[0036] Further, the basic language model is Qwen2VL-7B model.

[0037] The application also provides a knowledge reasoning device for integrated circuit wafer processes, comprising one or more processors for implementing the knowledge reasoning method for integrated circuit wafer processes as described above.

[0038] The application also provides a readable storage medium having a program stored thereon, which, when executed by a processor, implements the knowledge reasoning method for integrated circuit wafer processes as described above.

[0039] Compared with the prior art, the application has the beneficial effects that: the application proposes a brand-new multi-modal large language model for detection, retrieval and question-answering tasks in the integrated circuit field. The model gradually realizes the dynamic decomposition and high-precision solution of complex queries through three core stages. The introduced multi-modal chain reasoning framework (CoT) enables the model to combine logical reasoning with multi-modal information, and constructs a coherent solution chain to cope with complex tasks. In addition, the enhanced hybrid retrieval strategy combined with the adaptive ranking method effectively improves the relevance of the documents and the accuracy of the ranking. The application also fine-tunes the special wafer corpus using the LoRA technology, which strengthens the model's ability to detect defects, analyze root causes, and answer questions about defect types, locations and quantities, while significantly reducing the information forgetting problem of the model in multi-round dialogue. The results show that the model method of the application performs excellently in detection, retrieval and question-answering tasks on the internal wafer dataset and multiple test benchmarks (including 100 defect-related questions and integrated circuit general questions), significantly better than existing multi-modal large language models. The application not only provides strong support for knowledge analysis in the wafer manufacturing and integrated circuit fields, but also provides new insights and important references for the application and research of multi-modal large models in the field of industrial intelligence. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 A flow chart of a knowledge reasoning method for an integrated circuit wafer process.

[0041] Figure 2 A structural schematic diagram of a knowledge reasoning device for an integrated circuit wafer process.

[0042] Figure 3 A wafer knowledge dialogue example diagram. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0044] As shown in Figure 1 A knowledge reasoning method for an integrated circuit wafer process, which is composed of four core steps of wafer defect detection, wafer knowledge retrieval, document sorting and query answer generation, adopts Qwen2VL-7B model as the basic language model, and each step plays an indispensable role in the running and cooperates to realize efficient and accurate wafer manufacturing defect detection and knowledge query. The simple invention framework step is explained as follows:

[0045] Step 1, wafer defect detection

[0046] The wafer defect detection step is the basis of the whole invention. In this step, the multi-modal features of the input query wafer image are extracted to detect the image surface defects, and finally the Mask image capable of intuitively presenting the defect area is generated.

[0047] In the wafer defect detection step, the present application uses a frozen pre-trained encoder to encode the input wafer image and the wafer ID information extracted by OCR, fusing visual features and text features into a multi-modal vector representation. This multi-modal representation can effectively combine the information advantages of images and texts, so that the present application can capture the subtle features of wafer surface defects. For example, for common particle contamination, scratches and film unevenness, the present application can generate accurate defect region masks, while providing quantitative defect indicators in wafer surface quality evaluation. These information can not only be used for real-time quality control, but also provide data support for subsequent process optimization. Subsequently, the present application uses a mask decoder containing four layers of upsampling to further decode the encoded information and generate accurate defect region masks, thereby providing high-quality input data for subsequent retrieval and analysis tasks.

[0048] Step 2, wafer knowledge retrieval

[0049] In the wafer knowledge retrieval step, the retrieval task is divided into multiple sub-steps according to the complexity of the user query. In each sub-step, the wafer image, defect mask information and wafer ID are dynamically integrated to search for related documents step by step. For example, when the user queries "how to reduce particle contamination in the lithography process", the present application will first locate the historical defect data related to the lithography step, and combine the mask information to determine the typical position and characteristics of particle contamination, and further retrieve the best process practices related to contamination control. This step-by-step dynamic retrieval strategy can not only effectively cover all key information of complex queries, but also optimize retrieval accuracy at each step to ensure high relevance of search results to user needs, especially in specialized and technical fields.

[0050] In the wafer knowledge retrieval step, due to the high complexity and specialization of the wafer knowledge base, traditional retrieval methods usually rely on direct matching of query content, but this is difficult to effectively capture the user's potential intent. Especially in the field of integrated circuit manufacturing, when performing root cause analysis, it is necessary to comprehensively use wafer images, process data and technical documents to locate the problem points affecting yield. To solve this bottleneck, we propose a "logical decomposition-cross-modal fusion-multi-round retrieval" chain retrieval method, which can accurately meet the user's individual needs by refining the query, integrating multi-modal information and building context semantics. The steps of the chain retrieval method are as follows:

[0051] Step 2.1, logical decomposition of queries: In the field of wafer manufacturing, user requirements often contain multiple intentions, and directly processing these complex queries can easily lead to information omission. To this end, the invention designs a special query reasoner, which uses Qwen2.5-14B to gradually decompose the user query Q into multiple sub-queries and form a solution chain to capture key details. In this stage, the invention fine-tunes the data set of queries of different complexity through prompt learning, so that the Qwen2.5-14B model can recognize whether the query needs to be decomposed and generate the corresponding sub-query q i ∈{q1,q2,...,q k},q i represents the i-th sub-query, k represents the number of all sub-queries, i.e. the length of the solution chain, and the generation of each sub-query q i is determined by the following formula:

[0052]

[0053] Where arg max represents the parameter value that makes the target function maximum, R i-1 is the answer obtained from the previous sub-query q i-1 , and P is the conditional probability distribution. For simple queries, the invention directly outputs the final answer; while for complex queries, the length k of the solution chain is dynamically adjusted to further optimize the logical plan. This strategy can effectively reduce the computational cost of simple queries while ensuring detailed coverage.

[0054] Step 2.2, cross-modal fusion: In scenarios involving visual reasoning, understanding the user's query not only requires analyzing its logic, but also needs to combine relevant multi-modal information. To this end, the invention designs a shared gating unit to evaluate the relevance of wafer images, generated masks, and extracted wafer ID information to each sub-query, and selects the most relevant modal information to fuse with the query vector . Specifically, after the frozen pre-trained encoder generates the picture vector v image , the mask vector v mask and the text vector v id , the three vectors are concatenated and mapped to a smaller hidden space through linear transformation, thereby realizing the comprehensive evaluation of multi-modal information, and the expression is:

[0055]

[0056] Where concat represents the operation of concatenating data along the specified dimension, W and b represent the weight matrix and bias term respectively. F represents the vector after v image , v mask and v id are concatenated, and Fh represents the final vector mapped by linear transformation.

[0057] Subsequently, the correlation between modalities is calculated by a self-attention mechanism, and then a Softmax is used to assign weights to filter out the most relevant modal information for subsequent reasoning. Finally, the multi-modal query vector representation v mq is calculated as follows:

[0058]

[0059] wherein, the i-th sub-query vector generated by the encoder of Qwen2VL-7B, T represents transposition, d Fh represents the dimension of the vector F h . Through this process, the model can effectively combine user queries and multi-modal features to improve the understanding of the query context.

[0060] Step 2.3, multi-round retrieval: In the multi-round retrieval phase, the present application retrieves relevant candidate documents from the wafer knowledge base based on logical decomposition and cross-modal fusion for each user query Q and its sub-query q i and multi-modal vector v mq . Each round of retrieval provides necessary external knowledge cues for generating intermediate answers. To improve retrieval performance, the present application uses a hybrid retrieval method that combines keyword-based retrieval (such as TF-IDF, BM25) and semantic-based retrieval (such as Faiss). In operation, the sub-query q i uses keyword matching to ensure precise alignment of terms, while the multi-modal vector v mq captures complex contextual relationships through semantic retrieval. This iterative strategy can optimize the global retrieval capability of the model and improve the accuracy of obtaining wafer-related documents.

[0061] Step 3, document ranking

[0062] In the document ranking step, further weighting and scoring of the retrieval results are performed based on knowledge retrieval. Through a scientific scoring mechanism (such as keyword matching weight based on BM25, semantic embedding similarity, and wafer defect category relevance), the present application can prioritize the retrieved documents and select the top three documents most relevant to the query. For example, when processing a query involving "ion implantation doping concentration optimization", the present application will preferentially select documents containing detailed doping concentration curves and related experimental results. This process effectively reduces information redundancy and irrelevant document interference, providing highly relevant input materials for the subsequent answer generation step, thereby improving the overall reliability and accuracy of the present application.

[0063] In the document sorting step, during the root cause diagnosis process of wafer manufacturing, quickly locating the most relevant process conditions or previous failure analysis documents is of great significance for improving the problem troubleshooting efficiency and yield optimization. Wafer manufacturing is the core link of integrated circuit production, involving complex processes such as lithography, etching, ion implantation, and thin film deposition. Any minor defect or parameter deviation may lead to a decline in chip performance or even failure. To achieve this goal, the present invention designs an adaptive weighted sorting method. By performing a relevance score on all candidate documents and preferentially selecting the document with the highest score as an external hint, the inference burden of the basic language model is reduced while ensuring the retrieval accuracy. This method generates the final sorting result by dynamically adjusting the weights of the retrievers and combining multiple scoring strategies. The specific implementation steps are as follows:

[0064] Step 3.1, multi-dimensional scoring strategy for candidate documents: For each sub-query q i and its multi-modal query vector v mq , the present invention simultaneously uses three retrievers (TF-IDF, BM25, and Faiss) to extract a candidate document set D = {d1, d2,..., d n} from the knowledge base, where D represents the candidate document set, d1...d n represents the candidate documents, n represents the total number of candidate documents, and finally assigns a corresponding score to each document. The following are the scoring methods of each retriever: The TF-IDF score evaluates the document relevance based on the importance of terms, and the calculation formula is:

[0065]

[0066] where S t represents the TF-IDF score, t represents a term in the sub-query q i , IDE(t) is the inverse document frequency of this term, and f(t, d i ) is the frequency of the term t in the document d i . This method is suitable for exact matching of queries containing explicit terms.

[0067] The BM25 score optimizes the relevance between the query and the document by combining the term frequency and the document length. Its formula is as follows:

[0068]

[0069] where S b represents the BM25 score, avgdl is the average length of the documents in the corpus, and k1 and b1 are adjustment parameters, usually set to 1.5 and 0.7 respectively. The BM25 method smooths the high-frequency terms in long documents and is suitable for scenarios with a long corpus or unstructured data.

[0070] The Faiss score is based on the cosine similarity of vectors, evaluating the semantic relevance of the multi-modal vector representation to the document, whose formula is as follows:

[0071]

[0072] where S f represents the Faiss score, v mq represents the multi-modal query vector, and is the vector representation of the document d i . The Faiss score is particularly suitable for capturing complex contextual relationships and semantic information.

[0073] Step 3.2, semantic density calculation and retrieval adapter weight self-adaptive adjustment: To dynamically evaluate the semantic complexity of the query, the invention calculates the semantic density f i (q sem ) of each sub-query q i , which is defined as:

[0074]

[0075] where cosine represents the cosine similarity, and are the vector representations of the terms t1 and t2. The semantic density reflects the degree of semantic similarity between terms in the sub-query, and for complex queries, its value is usually high. Based on the semantic density f sem (q i ), the invention self-adaptively adjusts the weight vectors W t (TF-IDF), W b (BM25), and W f (Faiss) of the three retrievers, thereby achieving dynamic balancing of the scores. The calculation formula of the final score is:

[0076]

[0077] where β is the balance coefficient, set to 0.6. For semantically complex queries, the semantic retrieval weight of Faiss will be amplified, while for simpler queries, the weights of TF-IDF and BM25 will dominate.

[0078] Step 3.3, accurate re-ranking of candidate documents: In the final ranking stage, the invention re-ranks the candidate documents according to the final score S, and selects the most relevant documents from them. In this way, the invention can eliminate irrelevant or redundant content and only keep high-quality documents that are actually meaningful to the query. This not only improves the accuracy of knowledge retrieval, but also significantly reduces the reasoning pressure of the underlying language model.

[0079] Step 4, query answer generation

[0080] The query answer generation step is the final link of user interaction with the system. In this step, the invention inputs wafer images, defect masks, and sorted high-relevance documents into the base language model as prompts. The base language model generates accurate and explanatory answers by combining multi-modal data and professional document information to meet the user's professional query needs in the wafer manufacturing field. For example, when the user asks "how to solve the problem of increased wafer scratches in the chemical mechanical polishing process", the invention not only outputs detailed cause analysis (such as uneven pressure distribution or too large polishing liquid particles), but also provides specific improvement suggestions (such as adjusting the hardness of the polishing pad or optimizing the polishing liquid formula). In addition, the design of this step can generate detailed reasoning logic through additional reasoning processes, such as simulation effect prediction of different process parameter adjustments, so that the generated answers perform excellently in technicality and practicality.

[0081] In the query answer generation step, to meet the highly specialized needs of the wafer manufacturing field, the invention fine-tunes the Qwen2VL-7B model using LoRA technology and customizes it into a dedicated multi-modal large language model (MLLM) based on a multi-modal wafer corpus. During fine-tuning, the invention extends the input mode according to the characteristics of wafers in the integrated circuit field: first, the mask vector extracted from the query image is used as visual input to enhance the model's accurate understanding of the defect area; second, the top three knowledge documents highly relevant to the query are attached to the user query as additional text input, providing rich context support for the model. This input expansion design enables the model to better combine visual and textual modal information, thereby improving its ability to solve complex technical problems. During model adaptation, LoRA technology adjusts model parameters by introducing low-rank matrices, preserving the original model's powerful capabilities while significantly reducing the computational overhead of fine-tuning. After this optimization, the customized MLLM exhibits excellent performance in wafer manufacturing-related tasks, including high-precision defect detection, complex technical question answering, and root cause analysis, while being flexible in responding to diverse user needs. This field-specific model provides strong support for solving technical problems, significantly improving the knowledge management and problem diagnosis capabilities in the wafer manufacturing field.

[0082] Through the seamless cooperation of the above four core steps, the invention can efficiently solve complex query tasks in the wafer manufacturing field. Its technical advantages lie in accurate defect detection, comprehensive knowledge retrieval, reliable document sorting, and highly specialized answer generation capabilities.

[0083] Reference Figure 2The embodiment of the present application provides a kind of integrated circuit wafer process knowledge reasoning device, including one or more processors, for realizing the knowledge reasoning method of one kind of integrated circuit wafer process in the above embodiment.

[0084] The embodiment of the present application can be applied to any device with data processing capability, which can be a computer or other device.The device embodiment can be realized by software, hardware or a combination of software and hardware.Taking software implementation as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory and running by the processor of the device with data processing capability. Figure 2 As shown in the figure, it is a hardware structure diagram of the device with data processing capability of the present application, in addition to the processor, memory, network interface and non-volatile memory shown in the figure, the device with data processing capability in the embodiment usually includes other hardware according to the actual function of the device with data processing capability, which will not be described here. Figure 2 As shown in the figure, it is a hardware structure diagram of the device with data processing capability of the present application, in addition to the processor, memory, network interface and non-volatile memory shown in the figure, the device with data processing capability in the embodiment usually includes other hardware according to the actual function of the device with data processing capability, which will not be described here.

[0085] The implementation process of the functions and roles of each unit in the above device is described in detail in the implementation process of the corresponding steps in the above method, which will not be described here.

[0086] The technical features of the above embodiments can be combined arbitrarily, and to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of these technical features does not exist contradictory, it should be considered as the scope of the present application.

[0087] The embodiment of the present application further provides a readable storage medium, which stores a program, and the program is executed by a processor to realize the knowledge reasoning method of one kind of integrated circuit wafer process in the above embodiment.

[0088] The readable storage medium can be an internal storage unit of any data processing capable device in any of the preceding embodiments, such as a hard disk or a memory. The readable storage medium can also be an external storage device, such as a plug-in hard disk, a Smart Media Card (SMC), an SD card, a Flash Card, etc. equipped on the device. Further, the readable storage medium can also include both an internal storage unit of any data processing capable device and an external storage device. The readable storage medium is used to store the computer program and other programs and data required by the any data processing capable device, and can also be used to temporarily store data that has been output or will be output.

[0089] Embodiments

[0090] 1. Experimental data set: The internal wafer data set is randomly divided into a training set and a test set in a ratio of 7:3 for supervised defect detection training and performance evaluation. At the same time, the basic language model is trained using a multi-modal wafer corpus. To test the performance of the model in retrieval and question answering tasks, Qwen2-72B is used to generate 200 wafer defect related questions and 200 general IC related questions based on a knowledge database and a corpus. Then, 100 questions are selected from each type of question to form a test set. In the retrieval test set, each question is associated with three relevant documents, while in the question answering test set, each question is provided with a reference answer generated by a domain expert for performance comparison and verification.

[0091] 2. Experimental details: The frozen CLIP-L / 14@336px is used as the embedding model, which has powerful multi-modal feature extraction capability, and maps the image embedding to a high-dimensional feature space, providing a solid foundation for subsequent multi-modal information processing. At the same time, the fine-tuned Qwen2vl-7B is selected as the language generation model, which has excellent performance in text generation and multi-modal tasks, ensuring high accuracy and flexibility of the model in handling complex queries, technical question answering and specialized reasoning tasks. In the model training process, the efficient AdamW optimizer (β1=0.9, β2=0.999) is used, and the initial learning rate is set to 1e -4 , and then the cosine annealing scheduling strategy is used to gradually reduce the learning rate to 1e -6This optimization method can smoothly adjust the learning rate, avoiding overfitting while accelerating the convergence speed. In addition, combined with the LoRA fine-tuning strategy, the number of parameters and computational cost during model fine-tuning are greatly reduced, while preserving the original performance of the model and effectively adapting to the professional task requirements in the wafer manufacturing field. The model training is carried out on 4 A100 GPUs, and a total of 50 rounds of training are carried out, with each batch size set to 24. With such high-performance hardware support and refined training process, the model of the present application can efficiently capture key information in multi-modal data and exhibit excellent ability in tasks such as defect detection, knowledge retrieval and question answering in wafer manufacturing, providing important technical support for promoting the intelligentization of integrated circuit manufacturing.

[0092] 3. Experimental evaluation index: A multi-dimensional and multi-index method is used to comprehensively evaluate the performance of the model to ensure that its performance in detection, retrieval and generation tasks meets high standards.

[0093] In the detection task, five key indicators, including image-level AUC (Image-AUC), pixel-level AUC (Pixel-AUC), defect region proportion (PRO), average precision (AP) and accuracy, are used to evaluate the model. Specifically, ① Image-level AUC refers to the evaluation of the binary classification ability of the model for the entire image in the defect detection task. AUC (Area Under the Curve) is the area under the ROC curve, representing the model's ability to distinguish between positive and negative samples at all possible classification thresholds. Image-level AUC measures the accuracy of the model in identifying whether an image contains defects. ② Pixel-level AUC is used to measure the model's detection ability at a fine-grained level (i.e., pixel level). Unlike image-level AUC, pixel-level AUC focuses on the classification accuracy of each pixel within the defect region. It evaluates the classification ability of the model at each pixel point, especially suitable for boundary detection and precise positioning of defect regions. ③ Defect region proportion is used to measure the ratio of the defect region to the total image region when the model detects defects. This indicator reflects the coverage ability and positioning accuracy of the model for the defect region, and generally the higher the better, indicating that the model can better identify defects and avoid missing detection. ④ Average precision refers to the average value of the precision of the model at multiple thresholds, commonly used to evaluate the performance of object detection tasks. It considers the performance of the model at different confidence levels and calculates the precision at each recall rate, finally obtaining a comprehensive evaluation value. The higher the AP, the better the comprehensive performance of the model in the detection process. ⑤ Accuracy represents the proportion of correctly classified samples to the total number of samples. In the defect detection task, accuracy represents the proportion of correctly identifying defects or non-defects by the model.

[0094] In the retrieval task, three indicators of recall@n, normalized discounted cumulative gain@n (NDCG@n) and mean reciprocal rank@3 (MRR@3) are used to evaluate the retrieval ability of the model. Specifically, ① recall@n indicates the proportion of relevant documents in the nth result returned by the model in the retrieval task. That is, when the model returns only n results, the proportion of relevant results. ② Normalized Discounted Cumulative Gain@n measures the relevance of the nth retrieval result returned by the model. The normalized score is used to compare the effects of different models. This indicator considers the relevance and ranking of the results. The larger the value, the more accurate the ranking. ③ Mean Reciprocal Rank@3 refers to the average reciprocal of the position of the first relevant document in the first three retrieval results. This indicator reflects the speed of the model in retrieving relevant documents. The larger the value, the earlier the relevant document appears.

[0095] For the evaluation of the generation task, the invention introduces five indicators, including bilingual evaluation understanding score (BLEU), recall benchmark-1 (ROUGE-1), recall benchmark-longest common subsequence (ROUGE-L), word error rate (Wer), semantic text similarity (STS) and expert accuracy score (Expert Accuracy), to comprehensively measure the quality of the generated answers. Specifically, ① the bilingual evaluation understanding score is used to evaluate the quality of machine translation, which measures the n-gram overlap between the machine-generated text and the reference text. The higher the score, the closer the machine-generated text is to the reference text, which is suitable for text generation tasks. ② Recall benchmark-1 is used to evaluate the text summary or generation task, which calculates the word-level recall between the generated text and the reference text. ROUGE-1 mainly focuses on the recall ability of words, and the higher the score indicates that the generated text contains more words of the reference text. ③ Recall benchmark-longest common subsequence is used to measure the matching degree of the longest common subsequence (LCS) between the generated text and the reference text. Unlike ROUGE-1, ROUGE-L considers the sequence structure of the generated text, and the higher the score indicates that the generated text is better in maintaining the order and structure. ④ Word error rate is used to evaluate the accuracy of automatic speech recognition or text generation task, which calculates the edit distance between the generated text and the reference text, including the minimum number of insertions, deletions and replacements. The lower the word error rate, the closer the generated text is to the reference text. ⑤ Semantic text similarity measures the semantic similarity between two texts, which is suitable for text question and answer or text matching tasks, and the higher the score indicates that the two texts are more similar in semantics. ⑥ Expert accuracy score evaluates the correctness and logicality of the answers by artificial evaluation, providing the final professional judgment of the model generation results, ensuring the reliability and practicality of the generated content in the technical field. Through the comprehensive evaluation of the above multi-dimensional indicators, the performance of the model in different task scenarios can be fully grasped, providing scientific basis and direction for further optimization and improvement of the model.

[0096] 4. Experimental results:

[0097] (1) Test results:

[0098] Table 1 lists in detail the quantitative performance comparison results of the model method proposed in the application and other comparative models in five core indicators under the same training and testing conditions. From the table, it can be seen that the model method of the application shows obvious advantages in each indicator, especially in the three key indicators of image-level AUC, average precision and accuracy, which exhibit excellent performance improvement. Among them, the image-level AUC is improved by 1.62 compared with the latest model AnomalyGPT, which shows that the model method of the application is more accurate in overall detection effect; the average precision is improved by 2.23, which shows that it is more superior in the balance ability of precision and recall; and the accuracy is increased by 1.27, which reflects that the proportion of correct classification in all predictions is significantly improved, further verifying the reliability and adaptability of the model.

[0099] Table 1 Detection index results of the fully supervised experiment on the internal wafer dataset

[0100]

[0101] (2) Retrieval results:

[0102] The application uses the same wafer knowledge database and generated test set to comprehensively evaluate the performance of the four retrieval enhancement generation methods in the retrieval task, and focuses on comparing their performance in different indicators. The problems related to wafer defects usually focus on analyzing the causes of defects and formulating prevention strategies, and the following are two typical examples: 1) Please explain the cause of this defect; 2) How to prevent the occurrence of such defects? At the same time, the general problems related to integrated circuits (IC) cover the key links of process principles and manufacturing processes, and example problems include: 1) Briefly describe the principle of ion diffusion and its difference from thermal diffusion; 2) Why is it necessary to clean the wafer after polishing? 3) Why is wafer planarization necessary?

[0103] In the quantitative evaluation of retrieval performance, as shown in Table 2, the model method of the present application improved 0.107 in the recall@3 indicator compared with the BGE-M3 method. This result shows that the model method of the present application has higher coverage when retrieving relevant documents, can more effectively extract the information required by the user query, and significantly improves the breadth and accuracy of the content of the results. By retrieving a more comprehensive set of relevant documents, the model method of the present application provides users with more rich background knowledge support, especially in the context of complex technical queries and the combination of multi-modal data. In addition, as shown in Table 3, the model method of the present application also exhibits significant advantages in the sorting performance indicators. In the normalized discounted cumulative gain@3 indicator, the model method of the present application improved 0.127 compared with the BGE-Reranker, indicating that it has higher precision in the relevant document sorting process and can prioritize the most relevant documents in the front row of the retrieval results. At the same time, in the uniform ranking response rate@3 indicator, the model method of the present application improved 0.108 compared with the BGE-M3 method. This shows that the model method of the present application has significant advantages in optimizing relevant document sorting and improving the overall query result quality. The model method of the present application achieves comprehensive optimization from coverage to sorting accuracy in the retrieval task, not only ensuring the integrity of the information required by the user query, but also accurately identifying and arranging the most relevant documents. This performance advantage not only highlights the technical advancement of the model method of the present application, but also provides an efficient and reliable solution for knowledge retrieval and complex query tasks in the wafer manufacturing field.

[0104] Table 2 Retrieval indicator results on the wafer defect-related question and answer data set

[0105]

[0106] Table 3 Retrieval indicator results on the integrated circuit (IC) related question and answer data set

[0107]

[0108] (3) Question and answer results:

[0109] As shown in Tables 4 and 5, the present application conducted question and answer tests on different methods on two test sets, and the results showed that the model method of the present application showed significant advantages in multiple indicators. In terms of wafer-related questions, the model method of the present application is superior to the existing multi-modal large language model in most evaluation indicators. For example, in the recall rate benchmark-1 indicator, the model method of the present application improved by 0.093 compared with Vicuna-7B, which reflects its higher accuracy and stronger text generation ability in generating content similar to the reference answer. This shows that the model can more effectively capture key information and generate high-quality answers in the professional knowledge question and answer tasks in the wafer manufacturing field. For general questions of integrated circuits (IC), the model method of the present application is 0.410 higher than Qwen2VL-7B in the expert accuracy score, which further proves its excellent performance in technical and general tasks. The improvement of the expert accuracy score means that the model method of the present application not only can answer questions more accurately, but also can better meet the expert standard in logical reasoning and technical detail processing. This ability is particularly important for complex tasks such as process flow, design principles, and technical analysis related to integrated circuits.

[0110] Figure 3 Further, the model method of the present application demonstrates its dialogue performance in the wafer defect knowledge question and answer scenario. As can be seen from the figure, the model can accurately extract and use knowledge in the context of questions with technical depth, showing strong understanding ability and flexibility. This adaptability makes it perform more outstandingly and reliably in the question and answer tasks in the integrated circuit field, especially in handling complex technical problems. It can be seen that the model method of the present application not only performs excellently in professional questions related to wafer manufacturing, but also demonstrates high accuracy and universality in general technical question and answer of integrated circuits, providing a powerful solution for intelligent technical support in integrated circuit manufacturing and related fields.

[0111] Table 4 Question and answer index results on the wafer defect-related question and answer data set

[0112]

[0113] Table 5 Question and answer index results on the general integrated circuit (IC) related question and answer data set

[0114]

[0115] (4) Ablation experiment:

[0116] The present application conducts an in-depth ablation study on two different test sets (wafer defect and integrated circuit general problem) to comprehensively verify the effectiveness of the proposed module. In the experiment, by gradually removing or replacing the key modules in the model, the present application evaluates the specific contribution of each module to the overall performance. For the wafer defect test set, the focus of the study is on the influence of the module on the multi-modal information processing and the defect accurate identification ability; while in the integrated circuit general problem test set, the performance of the module in logical reasoning, knowledge retrieval and question and answer generation is evaluated. Through the ablation experiment, we can clearly quantify the role and influence of each module in different tasks, providing strong theoretical support and data basis for the design and optimization of the model.

[0117] ① Design of retrieval method: The present application conducts in-depth research on three different experimental conditions, namely: using fixed three-step query decomposition, not performing query decomposition, and completely not performing multi-modal information fusion. Through the experimental comparison of these conditions, the present application aims to evaluate the specific influence of each key link on the quality of answer generation. As shown in Table 6, the experimental results clearly show that dynamic query decomposition and fusion of additional information play a crucial role in improving the quality of answers.

[0118] Under the condition of fixed three-step query decomposition, although the model can generate answers of certain quality, its flexibility is obviously limited, making it difficult to deal with multi-layered intentions or potential details in complex queries. While under the condition of completely canceling query decomposition, the model's processing ability for complex problems decreases significantly, manifesting as important information omission and lack of answer completeness. At the same time, under the condition of not performing multi-modal information fusion, the model lacks comprehensive understanding of visual data and text data, and cannot fully utilize the advantages of multi-modal input, which directly leads to a significant decrease in the accuracy and relevance of the answers.

[0119] In contrast, when dynamic query decomposition and multi-modal information fusion are used, the model can intelligently decompose queries according to the complexity of the problem, gradually extracting and integrating key information related to the problem. This dynamic adaptability enables the model to perform better when dealing with complex problems, while the effective fusion of multi-modal information further enhances the model's understanding of the context and the accuracy of answer generation.

[0120] Overall, dynamic query decomposition and multi-modal information fusion not only enhance the flexibility and adaptability of the model, but also provide key support for generating more comprehensive and accurate answers. These experimental results fully demonstrate the technical advantages of the present application method in complex task processing, providing important reference for further optimizing the design of multi-modal models.

[0121] Table 6 Ablation results of CoT retrieval method on wafer defect-related and general integrated circuit question and answer data sets

[0122]

[0123] The effectiveness of the three rankers and the ranking calculation method is systematically verified through careful experimental design. The experiment aims to evaluate the performance of different rankers in retrieval results and analyze the improvement of overall efficiency and accuracy by the ranking calculation method. As shown in Table 7, the experimental results show that the ranking method proposed in the present application is superior to the average level of existing methods in many aspects, showing higher efficiency.

[0124] Specifically, the experiment compares the performance of the three rankers in handling queries of different complexity, including the accuracy of document relevance ranking, ranking efficiency, and support capability for retrieval tasks. The experimental results show that the proposed method not only can quickly complete the priority ranking of candidate documents, but also can dynamically adjust the weight during the ranking process to ensure that the most relevant documents are ranked at the top of the retrieval results. This ability is particularly significant in dealing with complex queries, significantly reducing the interference caused by irrelevant or low-quality documents. In addition, in the verification of the ranking calculation method, the comprehensive ranking mechanism proposed in the present application combines the results of multiple rankers to optimize performance. Compared with a single ranker, this method can more comprehensively utilize the multi-dimensional features of documents, such as keyword matching, semantic similarity, and multi-modal information fusion features. This multi-level ranking calculation method makes the final ranking result more accurate and significantly improves the efficiency.

[0125] It can be seen that the ranking method proposed in the present application achieves a good balance between efficiency and accuracy, overcoming the shortcomings of traditional ranking methods in high complexity tasks, and providing strong technical support for improving retrieval performance and optimizing user experience. The experimental results fully demonstrate the superiority and practical application value of the method, especially in handling complex technical document ranking tasks, which shows significant advantages.

[0126] Table 7 Ablation results on the question and answer dataset related to wafer defects and general integrated circuits on the retriever and ranking method

[0127]

[0128] Although embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and alterations can be made without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method of knowledge inference for integrated circuit wafer processes, characterized by, Comprise: Step 1, by extracting multi-modal features of the input wafer image, the image surface defect detection is carried out, and finally the mask image capable of directly showing the defect area is generated; Step 2, wafer knowledge retrieval is carried out, and in wafer knowledge retrieval, the retrieval task is divided into multiple sub-steps according to the complexity of the user query, and in each sub-step, the wafer image, defect mask information and wafer ID are dynamically integrated to complete the search of related documents step by step, including: Step 2.1, logical decomposition of query: design query reasoner, decompose user query into multiple sub-queries step by step, and form solution chain to capture key details, in this stage, through prompt learning, fine-tune the data set of different complexity queries, so that the basic language model can recognize whether the query needs to be decomposed, and generate corresponding sub-queries; For simple queries, directly output the final answer; for complex queries, further optimize the logical plan by dynamically adjusting the length of the solution chain; Step 2.2, cross-modal fusion: design a shared gating unit to evaluate the relevance of wafer images, generated masks and extracted wafer ID information to each sub-query, and select the most relevant modal information to fuse with the query vector; Step 2.3, multi-round retrieval: in the multi-round retrieval stage, based on logical decomposition and cross-modal fusion, the user query and its sub-queries and multi-modal vectors are retrieved from the wafer knowledge base to retrieve relevant candidate documents step by step; Each round of retrieval provides external knowledge prompts for generating intermediate answers; A hybrid retrieval method is used to combine keyword-based retrieval with semantic-based retrieval, in which sub-queries use keyword matching to ensure accurate alignment of terms, and multi-modal vectors capture complex context relationships through semantic retrieval; Step 3, document sorting, in the document sorting process, the retrieval results of wafer knowledge retrieval are weighted and scored, and the retrieved documents are prioritized through the scoring mechanism, and the top several documents most relevant to the query are selected; Step 4, generating query answers, in the process of generating query answers, wafer images, defect masks and high-relevance documents after sorting are input into the basic language model as prompts, and the basic language model generates accurate and explanatory answers by combining multi-modal data and professional document information.

2. The knowledge inference method of an integrated circuit wafer process according to claim 1, wherein, The step 1 includes: using a frozen pre-training encoder to encode the input wafer image and the wafer ID information extracted, and fusing visual features and text features into multi-modal vector representation.

3. The knowledge inference method of an integrated circuit wafer process according to claim 1, wherein, The step 2.2 specifically includes: The frozen pre-training encoder generates picture vectors, mask vectors and text vectors, which are spliced and then mapped to a hidden space through linear transformation, thereby realizing comprehensive evaluation of multi-modal information; Then, the correlation between modalities is calculated through self-attention mechanism, and then weighted by Softmax to select the most relevant modal information for subsequent reasoning, and finally, the multi-modal query vector is obtained.

4. The knowledge inference method of an integrated circuit wafer process according to claim 1, wherein, The step 3 specifically includes: Step 3.1, multi-dimensional scoring strategy of candidate documents: for each subquery and its multi-modal query vector, simultaneously extract a candidate document set from the knowledge base using the three retrievers of TF-IDF, BM25 and Faiss, and finally assign a corresponding score to each document; Step 3.2, semantic density calculation and retriever weight adaptive adjustment: calculate the semantic density of each subquery; Based on the semantic density, the weight vectors of TF-IDF, BM25 and Faiss are adaptively adjusted to achieve dynamic balance of the scores, and the final score is calculated according to the scores and weight vectors of TF-IDF, BM25 and Faiss, the semantic density, the balance coefficient; Step 3.3, accurate reordering of candidate documents: in the final ranking stage, the candidate documents are reordered according to the final score, and the most relevant documents are selected from them.

5. The knowledge inference method of an integrated circuit wafer process according to claim 1, wherein, The step 4 comprises: The LoRA technology is used to fine-tune the basic language model, and the basic language model is customized as a special multi-modal large language model based on the training of the multi-modal wafer corpus. In the fine-tuning process, the input mode is expanded according to the characteristics of the wafer in the integrated circuit field: Firstly, the mask vector extracted from the query image is taken as the visual input to enhance the model's accurate understanding of the defect area; Secondly, the top several knowledge documents highly related to the query are attached to the user query as additional text input, providing rich context support for the model; During the model adaptation process, the LoRA technology adjusts the model parameters by introducing a low-rank matrix.

6. The knowledge inference method of an integrated circuit wafer process according to claim 1, wherein, The basic language model is Qwen2VL-7B model.

7. An expert system for integrated circuit wafer processing, characterized by: The one or more processors are configured to implement the knowledge reasoning method of the integrated circuit wafer process according to any one of claims 1-6.

8. A readable storage medium, characterized by, The program is stored on the storage medium and is executed by the processor to implement the knowledge reasoning method of the integrated circuit wafer process according to any one of claims 1-6.

Images